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FOREWORD 


The  Simulation  Systems  Technical  Area  of  the  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  (ARI)  performs  research  and  development  in 
areas  that  include  training  simulation  with  applicability  to  military  training. 
Of  special  interest  is  research  in  the  area  of  simulation  fidelity  requirements. 

Before  any  training  system  may  be  developed  and  procured  for  use  in  the 
Army  training  community  its  specifications  must  be  determined.  These  training 
device  specifications,  when  compared  to  the  actual  equipment  may  be  defined  as 
simulator  fidelity.  It  is  necessary  to  determine  the  effects  of  level(s)  of 
fidelity  on  training  effectiveness  if  guidance  is  to  be  provided  to  support 
fidelity  decisions. 

This  report  presents  a  conceptual  framework  for  identifying  factors  that 
may  impact  training  simulator  effectiveness.  A  research  strategy  is  proposed 
for  the  empirical  determination  of  necessary  levels  of  training  simulator 
fidelity.  The  results  of  this  report  have  implications  for  PM  TRADE  and  for 
researchers  in  the  areas  of  training  and  training  device  development. 
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SPECIFICATION  OP  TRAINING  SIMULATOR  FIDELITY:  A  RESEARCH  PLAN 

BRIEF 


Requirement : 

To  develop  a  research  plan  that  can  guide  the  determination  of  the  empirical 
relationship  between  level  of  maintenance  training  simulator  fidelity  and  training 
effectiveness.  To,  furthermore,  ensure  that  this  plan  results  in  data  that  can  be 
used  to  provide  guidance  for  fidelity  specification  in  the  training  simulator 
development  process. 


Procedure : 

Three  activities  were  undertaken  at  the  outset  of  the  effort:  (1)  a  review 
of  the  empirical  literature  on  training  simulator  fidelity  was  conducted;  (2)  data 
collection  and  interviews  at  selected  Army,  Navy,  and  Air  Force  agencies  were 
carried  out;  and  (3)  a  contract  sponsored  workshop  entitled  "Research  Issues  in 
the  Determination  of  Simulator  Fidelity"  was  held.  A  conceptual  framework  to  guide 
this  and  subsequent  efforts  was  developed.  This  framework  identifies  and  defines 
factors  of  fidelity,  utilisation,  and  training  effectiveness  measurement  that 
impact  device  training  effectiveness.  A  research  strategy  was  created  based  upon 
considerations  Involving  (1)  operational  definitions  of  fidelity,  (2)  economic 
sequencing  of  studies,  (3)  theoretical  and  empirical  issues  of  maintenance  task 
training  and  required  skills,  and  (4)  desired  subject  population  characteristics. 
Preliminary  studies  consistent  with  this  strategy  are  proposed  to  explore  the 
effects  of  device  fidelity  on  the  transfer  of  training  of  perceptual -motor  and 
cognitive  maintenance  tasks. 


Findings: 

It  is  entirely  feasible  to  conduct  a  systematic,  empirical  investigation  of  the 
relationship  between  level  of  training  simulator  fidelity  and  training  effective¬ 
ness.  This  research  can  be  carried  out  in  such  a  way  as  to  provide  guidance  in 
the  context  of  the  training  simulator  development  process. 


Utilization  of  Findings: 

This  report  can  be  used  by  researchers  to  develop  specific  research  plans  for 
empirical  studies  to  determine  necessary  levels  of  training  simulator  fidelity  and 
by  the  military  trainer  development  community  to  establish  a  framework  for 
guidance  in  making  fidelity  decisions. 
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CHAPTER  I 


INTRODUCTION  AND  BACKGROUND 


PROGRAM  OBJECTIVES 

The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI) 
has  initiated  a  program  of  research  or.  guidelines  for  training  device  and 
simulation  development.  This  program,  known  as  SIMTRAIN,  has  three  major 
technical  objectives. 

1.  Evaluate  competing  methods  and  models  available  for  use-  in 
developing  and  evaluating  training  devices  and  determine 
appropriate  applications  in  the  existing  acquisition  process. 

2.  Develop  guidelines  for  relating  physical  and  functional  training 
device  characteristics  {i  -e.,  fidelity)  to  training  effective¬ 
ness,  with  a  focus  on  maintenance  training. 

3.  Evaluate  the  training  effectiveness  of  two  alternative  versions 
of  the  Army  Maintenance  Training  and  Evaluation  Simulation  System 
(AMTESS). 


PURPOSE  AND  ORGANIZATION  OF  THIS  REPORT 

This  report  presents  interim  findings  and  a  research  plan  to  initiate  empirical 
efforts  to  develop  data  in  support  of  the  second  objective  listed  above- 
specification  of  training  simulator  fidelity.  The  content  of  this  report  is 
based  on  a  review  of  the  relevant  literature,  data  collection,  and  interviews 
at  selected  Army,  Navy,  and  Air  Force  agencies,  and  the  results  of  a  contract 
sponsored  workshop  entitled  "Research  Issues  in  the  Determination  of  Simulator 
Fidelity." 
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Chapter  I  describes  these  data  collection  and  analysis  activities,  discusses 
issues  related  to  the  definition  of  fidelity  and  fidelity  specification,  and 
examines  additional  factors  that  determine  device  training  effectiveness. 

Chapter  II  presents  the  rationale  for  a  framework  within  which  a  specific 
plan  of  research  can  be  developed  and  conducted.  Chapter  III  presents  a 
recommended  series  of  studies  to  be  undertaken  within  that  framework. 

STATEMENT  OF  THE  PROBLEM 

It  is  widely  recognized  that  simulators  and  training  devices  offer  a  poten¬ 
tially  cost-effective  alternative  to  training  on  actual  equipment.  The  Army 
has  an  increasing  commitment  to  replace  or  supplement  hands-on  training  with 
training  simulators.  It  is  therefore  necessary,  in  order  to  realize  the 
potential  increases  in  cost-effectiveness  through  simulation,  to  establish  a 
systematically  and  empirically  derived  data  base  relating  training  simulator 
configuration  and  characteristics  to  training  effectiveness. 

Despite  the  many  years  of  R&D  on  flight  simulator  fidelity,  as  simulation  is 
applied  to  an  increasingly  wider  range  of  tasks  (e.g.,  equipment  maintenance), 
there  remain  questions  of  what  level  of  fidelity  is  appropriate,  when  to 
incorporate  instructional  features,  and  how  to  gain  user  acceptance. 

Fidelity  determination  is  a  central  issue  in  the  specification  and  design  of 
training  devices  and  simulators.  In  general,  the  issue  deals  with  the  degree 
of  fidelity  necessary  in  a  trainer  to  lead  to  a  given  level  of  training 
effectiveness.  Although  the  issue  appears  unitary,  it  actually  contains  many 
components  that  have  implications  for  all  steps  in  the  training  device  design 
and  acquisition  process,  and  for  the  overall  training  system  for  which  the 
device  is  designed. 

Considerable  progress  has  been  made  in  addressing  the  issues  surrounding  fidelity; 
however,  much  of  the  work  on  fidelity  has  been  aimed  toward  specific  areas  of 
interest  such  as:  particular  devices,  particular  interpretations  of  the  term, 
and  particular  models  designed  for  fidelity  determination.  The  work  has  so 
far  lacked  an  overall  programatic  effort  to  unify  the  concept  of  fidelity,  that 
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organizes  existing  work  and  indicates  direction  for  future  study.  In 
addition,  the  work  has  not  been  aimed  toward  its  application  by  the  service/ 
user  community,  and  has  not  developed  principles  and  guidelines  that  address 
the  particular  problems  surrounding  the  specification  of  fidelity  for  mainte¬ 
nance  trainers,  as  opposed  to  flight  or  team-trainers. 

The  current  effort  seeks  to  rectify  this  situation  through  a  program  of 
research  designed  to  examine  the  general  relationship  between  device  fidelity 
and  training  effectiveness  for  the  maintenance  task  domain.  This  report  de¬ 
scribes  a  plan  to  begin  a  programatic  empirical  research  effort  designed  to  pro¬ 
vide  data  upon  which  guidance  for  fidelity  dec is on -making  by  the  training  simu¬ 
lator  development  community  may  be  based.  The  methods  employed  to  help  achieve 
these  objectives  are  described  next. 


METHODS 

Three  major  activities  were  performed  during  the  initial  six  months  of  the 
contract.  These  activities  included: 

•  Literature  Review 

•  Site  Visits  and  Interviews 

•  Fidelity  Research  Issues  Workshop 

These  activities  are  described  in  the  following  paragraphs. 

Literature  Review 

The  objective  of  this  activity  was  to  create  and  then  analyze  and  synthesize 
a  bibliographic  data  base  including  design  guides,  technical  reports,  and 
academic  and  technology  journal  acticles.  The  review  emphasized  empirical 
as  opposed  to  theoretical  issues  of  training  device  fidelity  and  its  relation 
ship  to  training  effectiveness.  Both  computerized  documentation  services  and 
abstract  publication  lists  were  consulted. 
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The  documents  and  other  information  judged  relevant  to  this  effort  were 
obtained  and  cataloged.  A  review  of  the  documents  was  performed  and  annotated 
abstracts  prepared. 


Each  entry  into  the  data  base  contained  the  following  information: 
a  Internal  document  citation  number  (CN) 
a  Document’s  author 
a  Document's  title 
a  Source 

a  File  hardcopy  availability 
a  Abstract 
a  Keywords 

During  the  literature  review,  documents  were  abstracted  and  entered  into  a 
computerized  data  base.  This  data  base  is  set  up  on  Honeywell's  Multics 
computer  system.  During  the  present  SIMTRAIN  effort  additional  document 
entries  will  be  added  to  the  data  base. 

Entering  abstracts  into  the  Multics  data  base  offers  several  advantages.  The 
computer  can  perform  word  processing,  automated  storage  and  automated  retrieval. 
The  primary  advantage  of  the  data  base  comes  from  automated  retrieval  through 
interactive  queries.  Keyword  searches  can  be  conducted  in  a  number  of  ways, 
according  to  authors,  sources,  keyword  list,  or  even  keyword  phrases  contained 
in  each  abstract. 


The  information  accessed  in  the  search  is  printed  by  Multics  in  a  form 
dictated  by  the  searcher  (e.g.,  alphabetized  by  author's  last  name,  by 
consecutive  citation  number,  etc.).  In  addition,  at  any  time,  the  complete 
listing  of  all  information  in  the  data  base  can  be  printed  off  line  for  a 
complete  hard  copy  of  the  abstract  data  base. 
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The  abstract  data  base  is  intended  to  be  a  repository  for  information  relevant 
to  the  development  of  guidelines  for  training  simulator  fidelity  specification. 
It  will  be  documented  separately  and  updated  throughout  the  course  of  this 
research  program. 

Site  Visits  and  Interviews 


The  literature  review  effort  was  supplemented  with  site  visits  to  selected 
Army,  Navy,  and  Air  Force  agencies.  These  included: 

e  PM  TRADE . 

e  Army  Training  Support  Center  (ATSC), 
e  Naval  Training  Equipment  Center  (NTEC), 
e  Navy  Training  and  Analysis  Evaluation  Group  (NTAEG), 
e  Navy  Personnel  Research  and  Development  Center  (NPRDC),  and 
a  Air  Force  Human  Resources  Laboratory  (AFHRL). 

Discussions  covered  fidelity  determination,  training  effectiveness,  cost 
analysis,  front-end  analysis,  specification  development,  joint  working  group 
decision  making,  media  analysis,  training  delivery  systems,  individual 
differences,  advanced  technologies,  user  requirements,  user  acceptance,  con¬ 
current  prime/training  system  development,  and  test  and  evaluation. 

The  Army  training  device  development  process  is  summarized  separately  in 
Appendix  A  to  this  document. 

Fidelity  Research  Issues  Workshop 

In  order  to  develop  current  positions  on  the  research  issues  surrounding  the 
determination  of  simulator  fidelity,  an  invitational  workshop  was  organized 
and  conducted.  The  workshop  format  was  chosen  to  take  advantage  of  the 
expertise  of  the  participants  and  to  insure  that  issues  critical  to  Army 
training  requirements  were  addressed.  Invitations  were  extended  to  nearly 
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100  Individuals  representing  DoD,  Industrial,  and  academic  organizations. 

There  were  67  participants  from  16  DoD  agencies  and  16  Industrial  companies 
and  universities.  A  list  of  attendees  will  be  found  In  Appendix  B.  Volunteers 
for  presentations  were  solicited.  The  workshop  was  divided  into  presentation 
sessions  and  working  group  sessions. 

Presentation  Sessions— An  initial  presentation  session  provided  an  opportunity 
for  ARI  and  SIMTRAIN  contract  personnel  to  describe  the  associated  R&D  program, 
and  for  selected  participants  to  raise  issues  and  offer  perspectives  on  the 
problems  of  simulator  design  and  utilization.  A  closing  presentation  session 
permitted  the  topic  discussion  leaders  (see  below)  to  summarize  the  results 
of  the  working  group  sessions  and  selected  individuals  to  make  closing  observa¬ 
tions  and  final  remarks.  Transcripts  of  the  initial  and  closing  presentation 
sessions  will  be  published  in  a  separate  report. 

Working  Group  Sessions--The  working  group  sessions  were  structured  so  that 
each  group  had  an  opportunity  to  discuss  each  of  four  topic  areas: 

1.  Effectiveness  -  covered  issues  of  measurement  and  methodology 
in  training  and  cost  effectiveness  and  transfer  of  training. 

2.  Fidelity  -  covered  issues  of  measurement  and  methodology  in 
determining  the  relationship  between  physical /functional  training 
device  characteristics  and  effectiveness;  also  addressed  the 
general liability  of  previous  research  on  flight  simulators  to 
simulators  for  technical  (e.g.,  maintenance)  training. 

3.  Guidance  -  determined  appropriate  formats  and  contents  of  guidance 
for  fidelity  specification  decision  making;  also  assessed  when,  how, 
and  why  to  incorporate  emerging  technologies  into  simulators  and 
training  devices. 

4.  Priorities/Support  -  dealt  with  issues  of  topics  1-3  in  terms  of 
their  priorities;  also  addressed  the  format  and  contents  of 
agreements  needed  between  R&D  organizations  and  the  sponsor  and 
user  communities  to  promote  acceptance  of  R&O  studies. 
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Each  topic  area  had  a  topic  discussion  leader  and  an  archivist;  each  group  had 
a  group  leader.  The  groups  sequenced  through  the  topics  so  that  only  one 
group  discussed  a  particular  topic  at  one  time.  The  initial  working  group 
session  was  three  hours  in  duration,  with  the  three  subsequent  sessions  being 
one  and  one-half  hours.  Initial  working  group  assignments  matched  individual 
interests/expertise  to  a  topic  area  as  closely  as  possible.  The  topic  leaders 
facilitated  intra-group  coirmuni cation. 

The  format  chosen  for  the  working  group  sessions  was  successful  in  stimulating 
discussions  on  the  four  topic  areas.  A  wide  diversity  of  views  was  expressed 
as  to  the  appropriateness  or  form  of  the  specific  questions  assigned  to  each 
topic  area.  Indeed,  a  large  portion  of  Session  I  for  all  topics  was  devoted 
to  refining  or  restating  the  questions.  Therefore,  not  all  of  the  questions 
posed  at  the  outset  of  the  session  were  discussed  or  answered.  In  some  cases 
better  (i.e.,  more  meaningful)  questions  were  the  result.  The  main  points 
arising  from  discussions  of  each  topic  group  are  summarized  in  Appendix  B. 

Chapter  I  concludes  with  a  discussion  of  the  fidelity  concept  and  factors 
capable  of  influencing  simulator  training  effectiveness.  This  background 
is  based  on  an  integration  of  the  results  of  the  foregoing  activities.  It  is 
against  this  theoretical /conceptual  background  that  we  will  develop  a 
research  framework  and  plan  in  Chapters  II  and  III. 

FACTORS  IMPACTING  DEVICE  TRAINING  EFFECTIVENESS 

The  initial  objective  of  the  contract  effort  is  to  empirically  determine  the 
relationship  between  device  fidelity  and  training  effectiveness.  This  research 
must  be  carried  out  within  a  conceptual  framework  that  will  permit  the  devel¬ 
opment  of  guidance  related  to  the  many  factors  impacting  device  training 
effectiveness.  Before  substantial  progress  may  be  made  in  providing  such 
guidance,  the  numerous  variables  which  interact  with  simulator  fidelity  to 
produce  given  levels  of  training  effectiveness  must  be  empirically  investigated. 
However,  the  empirical  investigation  of  these  interactions  is  beyond  the 


scope  of  the  research  described  In  this  plan.  It  is  the  intent  of  the  research  j 

described  in  this  plan  to  construct  a  framework  which  will  investigate  the  j 

general  relationship  between  simulator  fidelity  and  training  effectiveness  j 

and  which  will  also  afford  a  means  for  the  systematic  accumulation  of  data 

on  all  interactive  variables  which  affect  this  relationship.  ! 

i 

The  basic  form  of  a  candidate  model  includes  three  factors:  . 

•  Fidelity  j 

J 

•  Utilization 

! 

i 

•  Training  Effectiveness  Measurement 

i 

Such  a  preliminary  model  provides  a  framework  for:  (a)  more  rigorous  modeling 
efforts;  (b)  research  efforts  addressing  the  value  of  the  included  factors, 
or  addition/deletion  of  factors;  and  (c)  measurement  and  validity  issues  involv¬ 
ing  various  aspects  of  training  simulator  effectiveness.  These  three  factors 
are  discussed  next. 

FIDELITY 

Fidelity  is  a  term  long  associated  with  training  devices  and  their  design 
(c.f.,  Miller,  1954).  Fidelity  has  the  status  of  an  explanatory  construct  in 
attempts  to  account  for  or  predict  the  training  value  of  a  device. 

We  next  briefly  review  definitional  issues  regarding  fidelity  and  select  a 
definition  for  present  purposes.  Various  theoretical  perspectives  on  the 
relationship  between  fidelity  and  transfer  of  training  are  subsequently  dis¬ 
cussed.  Finally,  variables  affecting  the  specification  of  fidelity  are 
described. 

Definition  of  Fidelity 

There  is  a  clear  lack  ofconsensus  in  the  literature  on  how  to  define  simu¬ 
lation  fidelity;  this  was  evident  at  the  workshop  as  well. 
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In  general,  definitions  of  fidelity  suffer  fro*  the  following  deficiencies: 

o  The  definitions  are  ambiguously  stated,  so  that  Investigators 
use  either  different  terms  to  describe  the  same  type  of  fidelity, 
or  the  same  terms  to  describe  different  types  of  fidelity. 

o  The  definitions  are  not  user-oriented;  they  give  little  guidance 
to  the  user  on  how  fidelity  may  be  determined  or  measured 
(Hays,  1980). 

A  major  step  in  the  development  of  a  scientific  approach  is  the  definition  of 
the  concept  of  interest.  Typically,  definitions  are  initially  informal,  and 

are  refined  as  additional  data  are  gathered.  Often,  a  definition  of  a  concept 
such  as  fidelity  must  best  be  viewed  as  a  "working"  definition--admittedly 
lacking  desired  precision,  but  which  can  serve  as  an  initial  ground  for 
generating  hypotheses  to  refine  the  concept. 

A  working  definition  of  fidelity  must  contain  at  least  the  following  three 
components: 

•  Fidelity  must  be  defined  in  terms  of  a  domain  of  interest  (X). 

e  Fidelity  must  be  defined  relative  to  something  else  (Y). 

% 

a  Fidelity  must  be  defined  so  as  to  be  measurable. 

A  definition  of  fidelity  must  therefore  be  of  the  form  "fidelity  of  X 
relative  to  Y  as  measured  (or  indicated)  by  Z  procedure.”  If  these  compon¬ 
ents  are  present,  then  a  working  definition  for  fidelity  can  be  formed. 

The  concept  of  fidelity,  in  itself,  is  not  the  major  issue.  Rather,  it  is 
the  degree  and  type  of  fidelity  of  training  devices  relative  to  operational 
equipment  that  is  required  to  lead  to  a  given  training  outcome.  Fidelity 
itself  may  be  a  superficial  concept;  as  Mstheny  (1978)  points  out,  a  dictionary 
definition  of  fidelity  means  merely  "duplication."  The  more  significant 
issue  is  the  departures  from  fidelity  that  can  be  undertaken  in  a  simulator 
which  will  lead  to  a  particular  level  of  performance.  Although  strict 
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duplication  can  occur  in  only  one  way,  departures  from  strict  duplication 
can  occur  along  many  dimensions.  Moreover,  departures  from  absolute  fidelity 
on  one  dimension  do  not  necessarily  covary  with  departures  on  other  dimen¬ 
sions.  Most  definitions  of  fidelity  that  have  been  generated  in  the  training 
conmunity  have  been  stated  In  such  a  way  so  as  to  emphasize  either  those 
dimensions  which  must  be  maintained^  to  leave  the  outcome  of  the  training 
unaffected,  or  especially  in  the  case  of  multidimensional  definitions,  those 
dimensions  which  can  depart  from  duplication  relatively  independently. 

Unfortunately,  these  definitions  have  often  been  confusing  and  contradictory 
among  themselves,  and  lack  the  precision  necessary  to  offer  the  user  practical 
guidance. 


The  uses  of  the  term  "fidelity"  have  been  numerous.  In  the  area  of  trainers, 
two  early  uses  of  the  term  tend  to  accompany  many  of  the  later  refinements  of 
the  term.  R.  B.  Miller  (1954)  introduced  the  term  "engineering  fidelity" 
to  describe  the  degree  to  which  a  trainer  duplicates  the  physical,  functional, 
and  environmental  conditions  of  the  prime  system.  This  purely  hardware 
definition  of  fidelity  is  not  necessarily  adequate  to  encompass  the  training 
demands  on  the  simulator.  Consequently,  the  term  "psychological  fidelity" 
was  introduced  (Gagne,  1954)  to  represent  the  trainee’s  perception  of  the 
"realism"  of  the  simulator.  A  distinction  here  is  that  the  trainee  may 
perceive  a  system  that  departs  significantly  from  duplication  as,  nevertheless, 
highly  realistic.  The  dimensions,  therefore,  may  be,  to  some  degree,  uncor¬ 
related.  The  distinction  between  these  two  types  of  fidelity  is  still  present 
in  contemporary  formulations.  Freda  (1979)  virtually  duplicates  these  con¬ 
cepts  in  his  two-component  definition  of  fidelity.  Physical  fidelity  is  the 
"engineering  representation"  of  the  operational  equipment,  whereas  psycholog¬ 
ical  fidelity  is  indicated  by  behavioral  and  informational  processing  demands 
on  the  trainee.  Freda  points  out  that  these  are  to  be  assessed  Independently, 
although  both  should  be  assessed  in  each  device. 

Many  definitions  of  fidelity  incorporate  aspects  of  both  engineering  and 
psychological  fidelity  in  their  formulation.  The  classic  three-component 
definition  of  fidelity  of  Kinkade  and  Mheaton  (1972)  Includes  1)  equipment 
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fidelity,  which  is  the  degree  to  which  the  simulator  duplicates  the  appear¬ 
ance  and  control-feel  of  the  operational  equipment;  2)  environmental  fidelity, 
which  is  the  degree  to  which  sensory  stimulation  (excluding  control-feel)  is 
duplicated;  and  3)  psychological  fidelity,  which  is  the  degree  to  which  the 
trainee  perceives  the  simulator  as  duplicating  operational  equipment.  Both 
equipment  and  environmental  fidelity  are  defined  in  terms  of  psychological 
variables,  such  as  Mfeel“  and  "stimulus  response"  conditions,  reflecting  the 
importance  of  the  training  process  in  determining  these.  (It  should  be  noted 
here,  as  in  much  of  the  work  on  fidelity,  that  the  definitions,  although 
meant  to  convey  general  principles,  are  drawn  mainly  from  flight  simulation, 
and,  consequently,  may  contain  features  inappropriate  to  maintenance  simulation. 
In  the  above  formualtion,  excluding  control -feel  from  environmental  fidelity 
is  relevant  in  the  sense  that  the  flight  simulator's  controls  are  separate 
from  the  through-windscreen  visual  display.  For  maintenance  training  simula¬ 
tors,  this  distinction  may  be  unnecessary  and  confusing.) 

Condon,  Ames,  Hennessy,  Shriver,  and  Seeman  (1979)  and  Fink  and  Shriver  (1978) 
offer  formulations  that  incorporate  both  engineering  and  psychological 
fidelity.  They  use  the  term  "physical  fidelity"  to  describe  layout  and 
"feel"  correspondence,  and  "functional  fidelity"  to  describe  stimulus/ 
response  correspondence.  These  dimensions  are  much  like  the  Kinkade  and 
Wheaton  (1972)  dimensions,  except  that  they  appear  to  contain  a  more 
"psychological"  aspect;  that  is,  both  physical  and  functional  fidelity 
are  described  in  terms  of  the  trainee's  perception  of  them. 

Condon,  et  al . ,  point  out  that  these  can  vary  independently  from  each  other, 
especially  when  type  of  task,  sequences,  or  groups  of  task  sequences  differ. 

That  is,  depending  upon  which  tasks  are  to  be  taught,  physical  and  functional 
fidelity  may  be  required  to  assume  different  values.  The  relevance  of  tasks 
is  addressed  also  by  Mirabella  and  his  colleagues  (Mirabella  and  Wheaton, 

1975;  Wheaton,  Mirabella,  and  Farina,  1971);  they  introduce  the  term  "task 
fidelity"  to  indicate  task  correspondence  between  the  trainer  and  the  opera¬ 
tional  equipment.  This  concept  is  very  much  like  "behavioral  fidelity" 

^  ■* 
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mentioned  by  Condon,  et  al .  The  term  "behavioral  fidelity"  may  be  preferred 
in  some  instances  to  an  overall  "psychological  fidelity,"  since  it  is  more 
descriptive  of  the  highly  behavioral,  ISD-based  approach  to  training  that  is 
presently  demanded  in  today's  military. 

These  conceptions  of  fidelity,  in  general,  provide  little  guidance  as  to  how 
they  should  be  assessed  in  practical  situations.  Device  design  guide 
developers  (Hirshfeld  and  Kochevar,  1979;  Miller,  McAleese,  Erickson,  Klein, 
and  Boff,  1977;  Marcus,  Patterson,  Bennett,  and  Gershan,  1980),  modellers 
of  predictive  training  effectiveness  for  device  design  (Swezey  and  Evans, 
1980;  Evans  and  Swezey,  1980),  and  manufacturers  of  training  devices  have 
chosen,  in  some  cases,  to  avoid  the  term  "fidelity"  in  favor  of  terms  that 
are  more  closely  related  to  the  actual  design  process.  Thus,  physical 
similarity,  functional  similarity  (Hirshfeld  and  Kochevar,  1979),  and  degree 
of  correspondence  of  cues,  responses,  and  actions  (Miller,  et  al . ,  1977) 
describe  fidelity-like  concepts  in  terms  more  closely  related  to  the  judgments 
that  must  be  made  in  device  design.  Physical  and  functional  requirements 
(Swezey  and  Evans,  1979)  emphasize  the  relationship  of  the  design  of  the 
trainer  to  the  training  requirements  which  must  be  met.  The  use  of 
fidelity-like  terms  for  the  practitioner,  rather  than  more  abstract 
definitions  of  fidelity,  may  aid  in  the  practical  work  of  trainer  design  and 
evaluation  by  not  adding  excess  meaning  of  "fidelity"  to  the  actual  actions 
and  conditions  that  must  be  met. 

Hays  (1980)  performed  a  literature  review  on  the  concept  of  simulator 
fidelity,  and,  in  general,  reached  the  following  conclusions: 

•  Many  concepts  have  been  used  to  describe  fidelity  in  simulators. 
Although  some  investigators  (e.g.,  Malec,  1980),  take  a  very  general 
approach  to  fidelity,  most  investigators  view  fidelity  as  consisting 
of  various  components. 

•  Investigators  disagree  on  the  important  aspects  of  fidelity.  The 
major  disagreement  seems  to  rest  on  the  difference  between  the 
fidelity  of  device  characteristics  ("physical"  or  practical  fidelity) 


12 


and  the  fidelity  that  the  trainee  experiences  ("physical"  vs. 
"behavioral"  fidelity).  Emphasis  on  each  of  these  may  lead  to  quite 
different  device  determinations. 


•  Investigators  have  developed  terms  for  fidelity  which  overlap  or  are 
inconsistent  with  each  other.  The  same  type  of  fidelity  is  called 
by  different  terms,  and  alternatively  different  investigators  use 
the  same  term  in  contradictory  ways. 

e  Definitions  are  rarely  operationalized  to  the  degree  that  they  can 
be  unambiguously  applied  in  specific  situations. 

•  Definitions  appear  too  "academic"  in  that  they  provide  little 
guidance  in  themselves  for  the  users  of  fidelity  concepts,  although 
guidebooks  have  been  developed  to  provide  "how  to"  information  for 
the  user. 

a  Fidelity  determination  must  take  into  account  issues  such  as  stage 
of  learning,  trainee  ability,  general  psychological  principles 
of  learning,  the  training  content,  and  type  of  task. 

Hays  (1980)  points  out  that  previous  definitions  of  fidelity  have  focused 
on  both  physical  device  characteristics  and  the  trainees'  perceptions, 
actions,  and  goals.  He  feels  that  these* should  be  separate:  "...fidelity 
should  be  restricted  to  descriptions  of  the  configuration  of  the  equipment 
and  not  be  used  when  discussing  behavior(s)"  (p.  14).  Fidelity  concepts 
"become  muddled  as  we  attempt  to  use  the  same  term  to  cover  all  aspects  of 
the  training  situation"  (p.  14).  Behavioral  and  psychological  implications 
of  fidelity  are  important,  but  they  should  be  described  in  such  terms  so  as 
to  not  confuse  them  with  device  fidelity.  The  latter  point  is  significant. 

It  is  necessary  that  fidelity  be  separated  from  the  psychological  implications 
of  fidelity.  This  may  be  especially  important  for  the  device  designer,  who 
is  necessarily  more  concerned  with  explicit  device  characteristics  than  with 
their  implications. 
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Specifically, Hays  (1980)  proposed  a  definition  of  fidelity  that  has  been 
subsequently  refined  as  follows  (personal  communication) : 

Training  simulator  fidelity  is  the  degree  of  similarity  between 
the  training  simulator  and  the  equipment  wiiich  is  simulated. 

It  is  a  two-dimensional  measurement  of  this  similarity  in  terms 
of: 

1.  The  physical  characteristics  of  the  simulated  equipment,  and 

2.  The  functional  characteristics  (i.e.,  the  informational 
or  stimulus  and  response  options)  of  the  simulated 
equipment. 

Fidelity  is  thus  a  descriptor  that  summarizes  the  overall  configuration  and 
characteristics  of  a  training  simulator.  All  device  features,  those  related 
to  the  actual  equipment  and  those  that  are  specific  to  the  device  (e.g.,  an 
instructional  capability),  impact  the  perceived  fidelity  of  a  training 
simulator.  So,  for  example,  a  device  specific  feature  that  reduced  fidelity 
in  this  sense  could  potentially  enhance  training  effectiveness  (Hays,  1980). 

It  is  this  definition  and  sense  of  fidelity  that  we  adopt  for  the  purposes 
of  the  present  program. 

Fidelity  and  Transfer  of  Training 

Various  basic  theoretical  approaches  have  been  suggested  to  describe  the 
relationships  among  fidelity,  transfer  of  training  and  other  related  variables. 
Several  of  these  recognized  approaches  are  presented  briefly  below. 

The  Skaggs-Robinson  Hypothesis  (Robinson,  1927)  was  an  early  attempt  to 
describe  the  relationship  between  transfer  of  training  and  similarity  of 
training  and  operational  equipment.  This  relationship  is  represented  by  a 
U-shaped  curve  where  positive  transfer  decreases  with  decreasing  similarity 
to  a  point,  after  which  transfer  becomes  negative  and  then  neutralizes  as 
similarity  continues  to  decrease.  The  implication  for  training  device 
effectiveness,  therefore,  is  that: 

•  High  similarity  results  in  high  positive  transfer; 
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•  Moderate  similarity  results  in  negative  transfer  (due  to 
confounding  effects);  and 

•  Low  similarity  results  in  essentially  zero  transfer. 

The  Osgood  (1949)  model  of  training  transfer  is  perhaps  the  best  known  model 
which  may  be  used  to  address  variances  in  amount  of  transfer  with  gradients 
of  similarity  between  the  operational  equipment  and  training  devices.  Osgood 
attempted  to  describe  this  relationship  using  a  three-dimensional  surface, 
relating  input  similarity  (S)  on  one  axis  and  output  similarity  (R)  on  the 
second,  to  amount  and  direction  of  transfer  on  the  third. 

According  to  the  Osgood  model,  with  identical  stimuli,  the  effect  of  variation 
in  required  responses  passes  from  maximum  transfer  at  identical  responses, 
through  zero  to  negative  transfer  as  antagonistic  responses  are  reached.  With 
identical  responses,  transfer  drops  to  zero  as  stimulus  similarity  decreases. 
On  the  other  hand,  with  antagonistic  responses,  transfer  rises  to  zero  from 
negative  as  stimulus  similarity  decreases. 


Critics  have  found  the  Osgood  model  to  be  deficient  in  predicting  negative 
transfer.  Bugelski  and  Cadwallader  (1956),  for  example,  in  testing  this 
model  obtained  very  similar  results  for  positive  transfer  predictions,  with 
contradictory  results  regarding  negative  transfer.  The  data  obtained  by 
Bugelski  and  Cadwallader  appeared  to  conform  much  closer  to  the  predictions 
based  on  the  Skaggs-Robinson  Hypothesis. 

Miller  (1954)  attempted  to  describe  the  relationships  between  simulator 
fidelity  and  training  effectiveness  in  terms  of  cost.  The  relationship 
posed  by  Miller  is  a  hypothetical  one  which  has  never  been  substantiated 
by  experimental  or  field  data. 

Miller  hypothesized  that  as  the  degree  of  fidelity  inthe  simulator  (training 
device)  increases,  the  cost  of  the  training  would  increase  as  well.  This 
relationship  is  represented  by  a  exponentially  increasing  "cost"  curve.  An 
S-shaped  "transfer"  curve  depicts  the  hypothetical  relationship  between 
fidelity  level  and  transfer  value.  At  low  levels  of  fidelity,  very  little 
transfer  value  can  be  gained  with  incremental  increases  in  fidelity.  However, 
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at  greater  levels  of  fidelity,  large  transfer  gains  can  be  made  for  small 
increments  in  fidelity.  The  curve  again  levels  off  at  high  levels  of  fidelity, 
where  further  increments  result  in  very  small  gains  in  transfer.  Miller 
hypothesized  a  point  of  diminishing  returns,  where  gains  in  transfer  value 
are  outweighed  by  higher  costs. 

Kinkade  and  Wheaton  (1972)  offered  several  general  guidelines  for  choosing 
among  alternative  training  device  designs.  The  most  significant  guideline 
in  their  model  highlights  the  non-absolute  nature  of  simulator  fidelity 
requirements.  Kinkade  and  Wheaton  (1972)  point  out  that  different  degrees 
of  equipment  and  environment  fidelity  (see  Definitions  of  Fidelity)  may  be 
appropriate  at  different  stages  of  training.  They  distinguish  among  three 
successive  stages:  procedures,  familiarization,  and  skill.  On  this  view,  a 
single  level  of  overall  fidelity  will  yield  differential  amounts  of  transfer 
depending  upon  the  stage  of  training. 

In  this  light,  it  is  important  to  point  out  that  assessments  of  the  physical 
and  functional  similarity  of  a  device  in  comparison  with  the  operational 
equipment  provide  only  an  indication  of  the  realism  of  the  training  equipment 
and  not  necessarily  its  training  effectiveness  potential.  There  are  a  Timber 
of  other  variables  that  determine  a  device's  training  effectiveness. 

According  to  the  Advisory  Group  for  Aerospace  Research  and  Development 
(AGARD;  1980)  for  example. 

Greater  value  could  be  put  on  this  measure  (fidelity)  if  we  could 
be  certain  that  high  fidelity  is  needed  in  every  case  in  order 
that  effective  training  and  transfer  of  training  can  take  place. 

However,  such  an  assumption  needs  to  be  treated  with  caution. 

First,  the  amount  of  fidelity  required  will  vary  with  the  nature 
of  the  task  to  be  trained,  and  so  training  effectiveness  is 
likely  to  vary  from  subtask  to  subtask  rather  than  be  represented 
by  one  unitary  value.  Second,  accepting  that  some  fidelity  is 
essential,  it  is  still  only  part  of  the  total  training  environ¬ 
ment.  The  purpose  for  which  the  device  is  used,  the  form  in  which 
the  training  is  given,  the  quality  of  the  instruction,  and  the 
attitudes  of  the  students  and  instructors  toward  synthetic  train¬ 
ing  will  all  influence  training  effectiveness.  Third,  high 
fidelity,  in  assessing  training  effectiveness,  is  sometimes  con¬ 
fused  with  the  needs  of  the  training  environment  itself.  This 


state  of  affairs  is  no  more  clearly  apparent  than  in  the  use  of 

large  simulators  in  commercial  aviation _ Training  may  be  possible 

with  far  less  sophisticated  devices.  Finally,  while  it  would 
appear  to  be  the  case  that  high  fidelity  generates  greater  user 
acceptance,  this  fact  does  not  of  itself  mean  that  a  device 
is  a  more  effective  training  facility  (p.  35). 

A  number  of  these  variables  are  or  should  be  taken  into  account  during  the 
training  simulator  development  process  in  which  levels  of  fidelity  are 
specified. 

Fidelity  and  Fidelity  Specification 

Fidelity  has  been  previously  defined  as  the  physical  and  functional  simil¬ 
arity  of  a  training  simulator  with  respect  to  the  actual  equipment.  A 
number  of  variables  affect  the  judged  level  of  device  fidelity  and  the  spec¬ 
ification  of  fidelity,  including  those  related  to  the  (1)  task(s)  to  be 
trained;  (2)  the  actual  equipment  characteristics;  and  (3)  the  requirements/ 
characteristics  of  the  training  environment  and  personnel. 

Task  Variables  and  Fidelity  Determination- -Most  authors  agree  (e.g..  Hays, 

1980;  Wheaton,  et  al.,  1976)  that  the  question  of  training  simulator  fidelity 
must  be  considered  in  the  context  of  particular  tasks  and  equipments.  The 
workshop  participants  concurred.  Of  particular  interest  is  the  influence  of 
task  variables  on  the  determination  of  fidelity  and  on  the  configuration  and 
characteristics  of  the  resulting  training  device  or  simulator.  A  brief  des¬ 
cription  is  provided  for  eight  task  variables  having  a  direct  impact  on  device 
design  and  thus  an  indirect  but  important  impacton  device  training  effectiveness. 


Task  Domain -^In  the  military  environment  where  personnel  interface  with 
equipment,  there  are  basically  two  task  domains:  operation  and  maintenance. 
Equipment  operation  may  be  further  divided  into  two  categories  depending  upon 
whether  the  equipment  is  moving  under  operator  control  (vehicles,  aircraft, 
guns,  and  so  on)  or  is  stationary  (display/control  consoles,  test  equipment, 
etc. ) . 


The  job  of  a  maintenance  technician  requires  that  he  or  she  also  be  able  to 
operate  equipment  (e.g.,  front-panels  in  the  conduct  of  built-in  tests). 

However,  rarely  does  the  equipment  operation  performed  by  the  maintenance 
technician  take  place  in  the  context  of,  for  example,  vehicle  control  (although 
a  test  ride  does  provide  a  counter  instance).  To  the  extent  that  different 
task  domains  ordomain  categories  require  different  degrees  of  fidelity  for 
effective  training,  the  task  domain  will  clearly  impact  fidelity  specification. 
Furthermore,  as  noted  earlier,  fidelity  may  have  a  different  definition  in 
the  context  of  a  task  associated  with  real-time,  real  world  views  and  movements 
than  in  that  of  a  task  performed  on  stationary  equipment. 

Task  Type--Perhaps  the  principal  factor  affecting  the  design  of  a  training 
simulator  is  the  match  between  the  type  of  task  being  trained  and  the  nature 
of  the  device.  While  one  could  enumerate  a  huge  variety  of  task  types  (c.f., 
Fleishman,  1967),  Card,  Moran,  and  Newell  (in  press)  have  offered  a  rather 
cogent  taxonomy  for  this  purpose.  They  postulate  the  existence  of  three 
principal  types  of  tasks:  sensory/perceptual  (those  involving  the  input  of 
information  to  the  human  processor),  cognitive  (those  involving  the  internal 
processing  of  the  observed  input),  and  motor  (those  involving  the  human's 
processed  output  and  their  translation  into  observable  actions).  The  design 
of  a  given  training  device  must  depend  in  large  part  on  the  extent  to  which 
it  must  allow  for  performance  of  a  particular  task  type. 

Task  Difficulty— This  variable  (see  Lenzycki  and  Finley,  1980,  for 
example)  relates  to  the  performance  difficulty  of  a  task.  It  reflects  both 
the  task  complexity  and  the  adequacy  of  the  work  environment  in  which  a  task 
is  to  be  performed.  According  to  Lenzycki  and  Finley,  there  are  four  levels 
of  task  difficulty:  unskilled  (requires  no  training  or  experience  to  accom¬ 
plish);  easy  to  perform  in  the  operational  situation  (equipment  and  work 
environment  adequately  designed;  normal  ambient  conditions  have  no  effect  on 
performance);  fairly  hard  to  perform  (some  constraints  in  the  operational 
envinonment) ;  and  hard  to  perform  (the  work  environment,  ambient  conditions 
or  the  equipment  design  can  produce  major  errors  in  task  performance). 
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Task  Frequency--This  factor  refers  to  the  frequency  with  which  a  task  is 
performed  under  operational  conditions.  Task  f requency  may  have  a  paradoxical 
effect  on  the  design  of  training  simulators.  While  a  frequently  performed 
critical  task  may  appear  to  warrant  a  high  cost  training  device,  the  amount 
of  practice  afforded  the  individual  in  the  operational  situation  may  offset 
this  requirement.  However,  the  infrequently  performed  critical  task  may  not 
be  as  frequently  practiced  and  may  require  the  more  costly  training  device. 

Task  Criticality— Task  criticality  refers  primarily  to  two  characteristic 
components:  (see  Cristal,  1973,  for  example)  delay  tolerance  and  consequences 
of  inadequate  performance.  Task  delay  tolerance  addresses  the  amount  of  delay 
which  can  be  tolerated  between  the  time  the  need  for  task  performance  becomes 
apparent  and  the  time  actual  performance  must  begin,  whereas  consequences  of 
inadequate  performance  measures  the  impact  of  human  error  on  the  system. 

Task  Learning  Difficulty-Another  training  simulator  design  variable 
involves  the  degree  of  learning  difficulty  associated  with  trainees'  attaining 
a  required  level  of  proficiency  for  a  particular  skill  or  knowledge.  Two 

task  characteristics  enter  into  such  an  analysis:  the  level  of  skill  or  know¬ 
ledge  proficiency  necessary  for  task  performance  and  the  level  of  learning 
difficulty  required  to  master  the  required  skills  or  knowledges  (see  Wheaton, 
et  al.,  1976).  As  an  example  of  the  variables,  four  levels  of  task  difficulty 
were  identified  by  Swezey  and  Evans  (1980):  easy  (trainee  can  accomplish  this 
activity  once  informed  that  it  exists;  virtually  no  practice  or  study  is 
required);  modestly  difficult  (trainee  can  accomplish  most  of  the  activity 
subsequent  to  instruction  with  little  practice  or  study,  but  some  of  the 
activity  does  require  minimal  practice  or  study  to  sustain  competent  perfor¬ 
mance  at  the  desired  level  of  prof iciency) ;  difficult  (trainee  can  accomplish 
the  activity  following  instruction,  but  only  with  consistent  practice  or 
study);  and  highly  difficult  (trainee  requires  extensive  instruction,  prac¬ 
tice  or  study  to  accomplish  the  activity;  requirement  at  least  borders  on 
expert  performance  standards). 
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Task  Practice  Requirements--Accordinq  to  Lenzycki  and  Finley  (1980), 
the  practice  requirements  of  a  task  is  the  first  criterion  against  which  one 
must  assess  tasks  to  determine  the  need  for  a  device  as  a  medium  for  training. 
This  variable  addresses  the  extent  to  which  initial  practice  and/or  sustain¬ 
ment  training  are  required  to  establish  and  maintain  an  acceptable  proficiency 
level  in  task  performance. 

Task  Required  Skills,  Abilities,  and  Know!  edges --Dunnette  (1976)  has 
recently  reviewed  the  literature  in  the  areas  of  human  skills,  abilities,  and 
knowledges.  The  establishment  of  what  types  of  skills,  knowledges,  abilities, 
etc.  are  required  by  various  tasks  to  be  trained  by  a  device  is  an  integral 
component  in  addressing  device  design.  For  example,  the  ease  with  which  tasks 
can  be  learned  in  a  device  and  transferred  to  operational  equipment  varies  with 
the  nature  of  the  task.  Procedural  skills  will  generally  transfer  readily 
but  will  be  forgotten  unless  practiced  regularly.  Perceptual -motor  skills 
transfer  less  completely  because  they  are  most  susceptible  to  imperfections 
in  the  simulation  of  dynamic  factors  of  environmental  fidelity  such  as  motion, 
visual,  and  kinesthetic  cues  and  control  forces.  Nevertheless,  while  the 
level  of  transfer  may  be  lower,  rapid  adaptation  appears  to  take  place  in  the 
operational  environment.  Devices  for  maintaining  procedural  skills  are  easier 
to  design  than  are  simulators  to  assist  in  the  retention  of  perceptual -motor 
Skill  (AGARD-AR-159,  1980). 

All  of  the  above  task  variables  must  be  considered  in  training  simulator 
design.  Typically  such  task  variables  are  considered  during  the  application 
of  task  analysis  techniques.  One  reason  for  reduced  training  simulator 
effectiveness  is  the  inadequacy  of  current  techniques  of  task  analysis  as 
applied  to  training  simulator  development.  An  analysis  of  the  tasks  to  be 
trained  must  result  in  a  clear  statement  of  what  knowledges  and  skills  have 
to  be  learned  (or  maintained)  for  job  proficiency.  There  are  several  differ¬ 
ent  task  analytic  frameworks  or  taxonomies  in  existence.  Each  one  tends  to 
have  been  designed  for  a  specific  purpose,  e.g.,  training,  operational  equip¬ 
ment  design,  simulation  model  development,  etc.,  or  from  a  particular  theor¬ 
etical  perspective,  e.g.,  behavioral  or  cognitive  (internal  or  mental 
mechanisms).  As  a  recent  experiment  demonstrated,  different  taxonomies  yield 
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significantly  different  results  in  simulations  based  on  task-oriented  network 
models  of  the  same  system  (Berry,  1980).  Clearly,  then,  there  is  the  strong 
possibility  that  training  simulators  based  on  task  analyses  utilizing 
different  taxonomies  will  yield  different  degrees  of  effectiveness.  Short¬ 
comings  of  a  task  analysis  will  promulgate  through  the  development  process 
and  ultimately  negatively  impact  force  readiness. 

Improved,  tailored  task  analytic  techniques  are  needed  for  the  training 
simulator  development  process. 

Actual  Equipment  Characteristics  and  Fidelity  Determination--Representation 
of  physical  and  functional  aspects  of  actual  equipment  in  a  training  simulator 
requires  detailed  knowledge  of  that  equipment.  That  detailed  knowledge  comes 
from  a  system  analysis  which  is  usually  carried  out  in  parallel  with  a 
task  analysis. 

The  functional  aspects  of  equipment  are  typically  more  difficult  to  ascertain 
than  the  physical;  this  is  especially  evident  in  simulation  of  systems  for 
maintenance  training.  For  example,  if  high  functional  similarity  is  determined 
to  be  necessary,  this  implies  that  a  working  simulation  ( i . e . ,  software)  model 
of  the  actual  equipment  must  be  developed.  Practical  experience  has  shown 
the  analysis  and  development  of  such  a  model  to  be  a  costly  and  time-consuming 
exercise. 

The  costs  of  achieving  high  physical  or  functional  similarity  can  be  extremely 
high.  To  achieve  cost  effective  training  we  must  be  able  to  specify  the 
minimum  required  level  of  fidelity  to  achieve  a  particular  training  purpose. 

Training  Environment/Personnel  Variables  and  Fidelity  Determination--There  are 
a  large  number  of  variables  associated  with  this  factor.  A  minimal  listing 
includes: 

•  Existing  or  projected  training  program  constraints 

•  Devi ce  purpose 

e  Instructional  principles 
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•  Student  population 

•  Instructor  population 

•  Safety 


Consideration  of  each  of  these  variables  will  affect  the  eventual  level  of 
fidelity  that  a  training  device  possesses.  Instructional  features  and 
student  population  are  briefly  covered  as  representative  examples. 

Instructional  Principles— Consideration  of  instructional  principles  in 
fidelity  determination  can  result  in  the  inclusion  of  instructional  features 
in  a  training  simulator.  Instructional  features  are  meant  to  enhance  or 
facilitate  the  learning  process,  yet  they  lower  the  fidelity  of  the  device 
because  their  incorporation  reduces  similarity  of  the  device  to  the  actual 
equipment.  These  features  include,  for  example,  augmented  feedback  or 
knowledge  or  results,  performance  measures  and  methods  like  adaptive  training. 

The  relative  contribution  to  training  effectiveness  of  instructional  features 
as  compared  to  actual  equipment  features  is  unknown.  Various  authors  and 
many  of  the  workshop  participants  claimed  that  instructional  features  can 
account  for  a  far  larger  portion  of  the  variability  in  training  effectiveness 
than  actual  equipment  features.  The  approach  adopted  in  the  SIMTRAIN  program 
is  that  this  is  an  empirical  question,  i.e.,  one  that  should  be  answered 
through  experiment.  This  research  plan  lays  the  foundation  for  this  and  other 
empirical  efforts. 


Student  Population--In  training  simulator  design  it  is  important  to  con¬ 
sider  individual  differences  in  aptitude,  ability,  or  ski  11 -level.  To  the 
extent  these  variables  are  taken  into  account  the  fidelity  of  the  resulting 
device  may  be  different.  For  example,  the  needs  of  low  aptitude  students  may 
result  in  the  incorporation  of  more  actual  equipment  features  or  functions, 
more  expensive  instructional  features,  or  both.  On  the  other  hanc,  the  needs 
of  high  aptitude  individuals  may  result  in  fewer  of  these  features  being 
incorporated  in  a  device.  Once  we  have  established  baseline  knowledge  of 
the  training  effects  of  fidelity,  we  can  begin  to  explore  the  potential 
interaction  of  fidelity  with  individual  differences. 


UTILIZATION 


This  is  the  least  well  understood  and,  according  to  some,  the  most  potent  factor 
in  determining  training  simulator  training  effectiveness.  A  model  of  utiliza¬ 
tion  factors  should  include  user  acceptance  and  motivation,  and  whether  or  not 
the  device  is  used  as  it  was  intended. 


The  factors  of  user  motivation  and  of  user  acceptance  of  training  devices  are 
of  major  interest  in  achieving  effective  training.  These  factors  have  been 
addressed  by  Burris,  et  al.,  (1571)  and  by  Glaser  and  Resnick  (1S72),  among  others. 
According  to  AGARD-AR-159  (1980),  device  effectiveness  is  known  to  be  highest 
when  an  instructor 


realizes  and  espouses  the  usefulness  and  relevance  of  the  device 
even  though  he  or  she  may  be  required  to  teach  around  faulty 
capabilities  or  features.  Students  tend  to  reflect  instructor 
attitudes.  Further,  where  recurrent  or  refresher  training  is 
being  provided  to  experienced  trainees,  with  or  without  instructor, 
an  element  of  competition,  or  a  comparative  such  as  a  probability 
of  success  in  combat  situations,  tends  to  motivate  trainees  to 
better  and  faster  learning,  (p.  9) 


A  recent  paper  by  Stoffer,  Blaiwes,  and  Bricston  (1980)  presents  a  preliminary 
model  of  the  acceptance  process  in  research  and  development.  A  number  of 
constraints  are  identified  that  can  work  against  user  acceptance  of  training 
R&D  studies  and  training  simulators.  These  are: 

1.  Deficiencies  in  user  motivational  conditions 

2.  Deficiencies  in  user  role  assignments 

3.  Deficiencies  in  official  policy  and  structure 

4.  Inadequate  defense  R&D  contracting  methods 

5.  Inadequate  integration  of  the  user  into  the  acquisition  process 
through  participative  management 

6.  Other  than  rational  user  responses  to  R&D  studies  and  to  training 
simulators 

7.  Deficiencies  in  training  simulator  design 
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Training  simulators  have  the  potential  to  train.  Students  and  instructors 
nust  use  training  simulators  in  such  a  way  as  to  maximize  that  potential. 
The  best  design  in  the  world  will  remain  just  that  unless  users  accept  a 
training  simulator  and  fully  incorporate  it  into  their  training  program. 


We  must  increase  our  understanding  of  how  user  acceptance,  for  example, 
impacts  training  simulator  effectiveness.  Increased  understanding  of  the 
problem  will  lead  to  an  ability  to  predict,  control ,  and  thus,  improve  it. 
As  Stoffer,  et  al . ,  (1980)  state: 

Some  may  resist  the  idea  that  science  can  be  used  to  analyze 
and  actually  influence  something  so  apparently  nebulous  and 
subjective  as  "acceptance."  Although  the  state  of  science  is 
not  well  developed  in  the  acceptance  area,  there  are  some 
theoretical  and  empirical  bases  for  influencing  levels  of 
acceptance.  An  initial  step  in  the  scientific  approach  would 
be  to  document  the  extent  of  variation  of  acceptance  found  in 
various  aspects  of  (naval)  training.  Factors  that  influence 
acceptance  would  subsequently  be  identified,  described  and 
prioritized  for  different  applications.  Then  improved  metrics 
for  these  factors  would  be  generated.  A  conceptual  framework 
consisting  of  these  factors  and  their  relationships  to  one 
another  would  be  developed  to  understand  the  process  of 
acceptance  and  to  use  as  a  basis  for  predicting  acceptance 
levels  in  particular  situations.  Some  of  these  factors 
have  predictable,  but  uncontrollable  consequences  on  acceptance 
levels;  other  factors  are  controllable  by  those  with  influence 
in  the  training  command  structure.  The  R&D  community  can 
become  more  proficient  at  managing  the  introduction  of  training 
innovations  by  applying  those  factors  that  can  be  controlled 
to  influence  acceptance,  (p.  19) 


The  same  approach  could  and  should  be  followed  to  deal  with  other  aspects  of 
the  utilization  issue.  Further  analytic  and  empirical  work  based  on  these 
authors'  model  appears  warranted. 


TRAINING  EFFECTIVENESS  MEASUREMENT 

The  previous  sections  describing  fidelity  and  utilization  covered  independent 
variables  that  can  influence  training  device  effectiveness.  What  dependent 
variable(s)  should  be  used  to  define  training  effectiveness?  Answers  to  this 
question  bear  ultimately  on  whether  the  simulator  is  viewed  as  resulting  in 
more  effective  training  and  improved  on-the-job  performance. 
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The  choice  of  training  effectiveness  measures  is  not  a  simple  one.  Typically, 
the  effect  of  the  training  device  on  eventual  performance  on  the  operational 
equipment  is  viewed  as  a  two-stage  process. 

•  Training  effectiveness  on  the  device.  This  is  the  degree  to  which 
training  is  accomplished  on  the  training  device,  and  is  assessed  by 
various  measures  of  improvement  on  the  device. 

•  Transfer  of  training  to  the  operational  equipment.  This  is  the 
degree  to  which  the  training  accomplished  on  the  device  generalizes 
to  performance  on  the  operational  system. 

In  most  cases  training  effectiveness  and  transfer  of  training  issues  are 
highly  related.  However,  instances  exist  where  the  two  should  be  treated 
separately.  For  example,  the  case  where  performance  on  the  training  device 
is  excellent,  but  contributes  little  to  performance  on  the  operational  equip¬ 
ment.  Another  possibility  is  where  a  device  may  be  unpopular  or  lack  face 
validity,  but  nevertheless  leads  to  excellent  performance  on  the  operational 
equipment.  Such  relationships  must  be  evaluated  in  the  conduct  of  a  program 
of  research  on  simulation  fidelity. 

In  general,  three  methods  exist  for  the  purpose  of  evaluating  training  simu¬ 
lator  effectiveness.  One  involves  a  transfer  experiment,  where  various  items 
of  training  equipment  are  compared  either  to  each  other  or  to  relevant 
operational  equipment  in  terms  of  transfer-of-training.  A  second  involves 
ratings,  where  various  experts  (for  instance,  pilots)  are  asked  to  rate 
training  devices  or  simulators  on  their  perceived  utility  for  training 
(see  Adams,  1979,  for  a  discussion  of  these  issues).  A  third  method  is 
the  development  of  analytic  models  such  as  the  TRAINVICE  family  of  models 
(see  Wheaton,  et  al.,  1976;  Swezey  and  Evans,  1980).  In  this  approach, 
analytic  techniques  are  employed  to  investigate  the  extent  to  which  simulators 
or  training  devices  adequately  cover  necessary  training  requirements,  address 
relevant  tasks,  and/or  provide  for  training  on  appropriate  skills,  knowledges, 
and  abilities. 
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These  methods  are  not  without  criticism.  Adams  0979),  for  example,  discusses 
shortcomings  of  both  the  transfer  of  training  and  ratings  methods  in  the 
context  of  flight  training  simulators.  Adams  himself  prefers  an  analytical 
approach  based  on  whether  or  not  the  training  device  is  designed  according 
to  reliable  scientific  laws  of  human  learning.  However,  this  is  somewhat 
circular  as  the  reliability  of  the  scientific  laws  is  dependent  upon  the 
demonstrated  success  of  other  devices  designed  according  to  the  same  principles. 

It  is  our  opinion  that  effectiveness  data  based  on  only  one  measurement  tech¬ 
nique  is  insufficient  for  research  purposes.  The  converging  evidence  provided 
by  using  two  or  more  of  the  methods  will  be  invaluable  to  assessing  the 
training  effects  of  device  fidelity. 

SUMMARY 

The  Army's  goal  of  achieving  cost-effective  training  through  the  use  of 
simulators  can  be  met  through  a  sustained  program  of  training  device  research. 
Because  high  fidelity  training  devices  are  costly,  data  are  needed  on  the 
relationship  between  fidelity  and  training  effectiveness.  Many  factors  impact 
fidelity  determination  and  training  device  effectiveness.  A  systematic, 
empirical  investigation  of  these  factors  is  necessary.  In  the  following 
chapters  we  describe  a  framework  within  which  this  research  can  be  carried 
out  (Chapter  II)  and  specific  studies  that  should  be  undertaken  (Chapter  III). 
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CHAPTER  II 


A  FRAMEWORK  FOR  FIDELITY  RESEARCH  IN 
MAINTENANCE  TRAINING 


The  objective  of  the  initial  phase  of  our  research  is  to  conduct  preliminary 
studies  on  the  relationship  between  fidelity  and  training  effectiveness  for 
maintenance  tasks.  Any  particular  study  should  be  carried  out  within  a  frame¬ 
work  providing  for  a  programmatic  approach.  The  framework  must  include  a 
definition  of  fidelity  that  can  be  operational ized  and  a  strateqy  for  explorinq 
various  levels  of  fidelity.  The  framework  must  be  based  upon  a  set  of  tasks 
representati ve  of  Army  maintenance  requirements  and  the  results  of  previous 

empirical  fidelity  research  with  such  tasks.  It  must  provide  a  structure  for 
the  accumulation  of  future  empirical  data,  such  as  the  effects  of  potentially 
interactive  variables.  Finally,  the  framework  must  consider  issues  related 
to  subject  populations.  In  this  chapter  we  develop  the  framework  that  will 
guide  the  specification  of  a  program  of  research  on  training  simulator  fidelity. 
The  research  described  here  is  designed  to  be  the  first  step  in  establishing 
the  required  framework  for  future  efforts. 

FIDELITY  AND  RESEARCH  STRATEGY 

Hays  (1980)  has  described  a  framework  for  examining  the  effect  of  fidelity 
on  training  effectiveness.  Following  his  distinction  between  physical  and 
functional  aspects  of  fidelity,  he  suggests  treating  these  as  separate  factors 
in  an  experimental  design.  Figure  I  shows  a  sample  design  within  this  framework. 

Operational  Definitions  of  Fidelity 

Physical  and  functional  similarity  may  be  operationalized  in  terms  of  two 
scales  (Wheaton,  et  al . ,  1971;  Wheaton  and  Mirabella,  1972;  and  Mirabella 
and  Wheaton,  1974).  A  panel  layout  index  (PLI;  Fowler,  et  al . ,  1968)  can  be 
used  to  determine  physical  similarity  of  a  training  simulator  as  compared 
to  the  actual  equipment.  Functional  similarity  can  be  assessed  using  the 
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Display  Evaluation  Index  (DEI)  which  measures  the  effectiveness  of  the 
display-control  information  flow  (Siegel,  et  al . ,  1962). 

A  shortcoming  of  this  approach  is  that  the  scales  apply  to  equipment  operation 
consoles,  e.g.,  automatic  test  equipment,  and  would  not  be  directly  applicable 
to  equipment  lacking  control  panels  and  displays,  e.g.,  a  mechanical  system. 

In  these  instances,  observer  judgment  would  provide  a  substitute. 

The  observer  could  be  asked  to  rate  physical  and  functional  similarity  on 
scales  where  each  scale  category  was  defined  in  terms  of  degree  of 
similarity.  Alternatively,  a  direct  magnitude  estimation  procedure  could 
be  used.  If  enough  alternatives  were  available,  paired  comparison  estimates 
of  overall  fidelity  could  be  obtained  and  the  results  analyzed  using  non-metric 
multidimensional  scaling  techniques  (Shepard,  1964).  In  this  manner  the 
dimensionality  of  the  space  characterizing  the  alternative  training  simulators 
could  be  empirically  determined.  According  to  our  current  definition  we 
would  expect  two  dimensions. 

Our  approach  will  be  to  define  fidelity  in  terms  of  a  PL1  and  DEI  or  observer 
judgment  of  similarity,  as  appropriate. 

Sequence  of  Studies 

A  potential  problem  with  the  design  depicted  in  Figure  1  is  that  the  two 
aspects  of  fidelity  may  not  be  entirely  independent.  For  example,  it  might 
prove  difficult  or  impossible  to  engineer  a  device  that  is  low  in  physical 
similarity,  while  at  the  same  time  being  high  in  functional  similarity.  In 
such  instances  we  would  have  to  extrapolate  from  conditions  that  could  be 
achieved. 
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Additionally,  achieving  particular  configurations  of  degrees  of  similarity 
could  prove  costly  and  the  cost  of  providing  nine  alternatives  might  outweigh 
the  benefit.  Therefore,  our  strategy  will  be  to  take  an  incremental  approach. 
Initially  we  will  explore  cells  1,  2,  and  3  (see  Figure  1)  where  high 
physical/high  functional  similarity  v<i 1 1  be  represented  by  the  actual 
equipment.  Based  upon  significant  effects  along  the  diagonal  we  can  then 
begin  disentangling  the  effects  of  physical  and  functional  similarity. 

This  will  be  done  by  filling  in  the  other  cells  beginning  with  4  and  5 
assuming  they  are  technologically  achievable. 

FIDELITY  RESEARCH  AND  MAINTENANCE  TASKS 

In  the  previous  section  we  developed  a  research  strategy  based  upon  considera¬ 
tions  of  how  to  operationalize  simulator  fidelity.  The  implementation  of  the 
strategy  and  the  instantiation  of  levels  of  training  simulator  fidelity  require 
the  context  of  a  specific  task  or  tasks.  We  must  have  some  criteria  for 
determining  which  task(s)  to  study.  In  this  section  we  consider  criteria 
relating  to  the  nature  of  equipment  maintenance  in  the  military,  to 
general izabi 1 ity  of  experimental  results  and  to  previous  empirical  research. 

In  Chapter  III  criteria  relating  to  the  suitability  of  tasks  for  laboratory 
research  are  discussed. 

The  basic  job  of  the  equipment  maintainer  is  to  fault  isolate  and  repair  specific 
systems.  Depending  upon  whether  the  system  is  electronic,  mechanical,  hydraulic, 
or  hybrid  in  nature  and  whether  the  maintenance  is  carried  out  at  the 
Organizational,  Oirect  Support,  General  Support,  or  Depot  level,  the  details 
of  the  job  tasks  change.  These  details  minimally  include: 

•  amour*,  of  troubleshooting  skill  needed; 

•  tools  or  test  equipment  utilized; 

•  degree  of  automatic  testing  and  fault  isolation  provided;  and 

•  actions  required  to  remove/ replace  components  or  adjust, 
align,  or  calibrate  them. 
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Given  the  differences  that  exist  among  maintenance  tasks,  how  can  we  select 
tasks  so  as  to  (1)  achieve  generality  and  (2)  maximize  knowledge  gained  from 
previous  research?  Both  of  these  objectives  can  be  met  if  we  accept  a  categori¬ 
zation  of  tasks  based  on  skills  required. 

It  is  generally  agreed  in  contemporary  psychology  that  there  are  two  broad 
categories  of  skill,  namely,  perceptual -motor  and  cognitive  (Fitts  and  Posner, 
1967;  and  Wei  ford, 1968) .  Both  of  these  categories  can  be  more  finely  divided. 
Perceptual -motor  skill  includes  sensory,  perceptual,  and  manipulative  components. 
This  category  subsumes  the  Card,  et  al .  (in  press)  sensory/perceptual  and 
motor  task  types.  Cognitive  skill  (Card,  et  al's.  third  task  type)  is  ex¬ 
emplified  by  language  comprehension,  computation,  decision-making,  and  problem¬ 
solving. 

The  performance  of  actual  military  maintenance  tasks  requires  a  combination  of 
these  skills.  However,  many  complex  tasks  can  be  decomposed  into  their  elemen¬ 
tal  components  and  upon  so  doing  they  can  be  seen  to  emphasize  one  type  of  skill 
as  compared  to  another. 

In  the  military  equipment  maintenance  environment  both  kinds  of  skills  are  im¬ 
portant.  Perceptual -motor  maintenance  tasks,  i.e.,  those  requiring  principally 
perceptual -motor  skills,  include  such  tasks  as  adjusting  a  carburetor,  aligning 
radar  synchronizers,  and  the  like.  Cognitive  maintenance  tasks  include  such 
tasks  as  interpreting  oscilloscope  waveforms,  troubleshooting  a  system  to  the 
component  level,  and  the  like. 

Other  tasks  require  the  integration  of  perceptual -motor  and  cognitive  skills. 

In  the  military  maintenance  environment  procedural  tasks  are  the  most  im¬ 
portant  example  of  this  integration. 
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There  are  thus  three  types  of  tasks  that  can  be  studied  in  the  context  of 
fidelity  research.  However,  two  issues  remain:  Which  tasks  do  we  already 
have  good  data  on?  What  should  our  priorities  be  in  selecting  tasks  to  study? 
The  following  discussion  provides  answers  to  these  questions. 

Procedural  Maintenance  Tasks 


Procedural  tasks  are  those  involving  a  preferred  or  proscribed  sequence  of  events 
and  actions.  Procedures  are  performed  either  by  following  technical  material, 
e.g.,  skill  performance  aids,  or  by  recalling  the  steps  from  memory. 

What  do  we  know  about  the  level  of  fidelity  required  to  train  tasks  that  are 
primarily  procedural?  Our  review  of  the  literature  indicated  that  we  know 
quite  a  lot.  Numerous  studies  performed  over  the  past  30  years  have  con¬ 
cluded  that  high  training  simulator  fidelity  is  not  necessary  for  the 
effective  training  of  procedural  tasks  (c.f.,  Bernstein  and  Gonzalez,  1971; 
Grimsley,  1969a,  1969b;  Cox  et  al . ,  1965;  Prophet  and  Boyd,  1970;  Mirabella 
and  Wheaton,  1974;  Crawford  and  Crawford,  1975;  and  Johnson,  1981).  These 
studies  have  compared  various  low-fidelity  training  devices,  e.g.,  photographs, 
charts,  mock-ups  and  flat-panel  simulators,  with  actual  equipment  and  found 
them  to  be  equally  effective.  It  thus  appears  that  physical  similarity  is  more 
important  than  functional. 


This  general  result  has  been  obtained  for  both  procedures  following  and  recalling 
primarily  in  the  context  of  vehicle  and  console  operation,  for  example,  start-up, 
check-out,  and  shut-down  procedures.  Such  procedures  typically  involve  switch 
setting,  button  pressing  and  the  reading  of  discrete  display  information.  To  the 
extent  that  equipment  troubleshooting  involves  procedures  for  equipment  operation, 
e.g.,  use  of  (automatic)  test  equipment  or  running  and  interpreting  built-in  tests, 
the  data  indicate  that  training  can  be  accomplished  using  low-cost  devices  with 
low  functional  similarity  and  moderate  to  high  physical  similarity. 
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Orlansky  and  String  (personal  communication)  have  noted  that  although  this 
evidence  cannot  be  denied,  it  has  not  had  a  major  influence  in  the  development 
of  maintenance  training  simulators.  Given  every  indication  that  this  type  of 
(front-panel)  maintenance  activity  for  electronics  systems  will  continue  to 
grow  (c.f.,  Fink  and  Shriver,  1978;  or  Kenney,  1977)  the  maintenance  training 
community  should  take  a  hard  look  at  this  deficiency. 


As  noted  earlier,  the  performance  of  any  particular  procedure  will  require  either 
or  both  perceptual -motor  or  cognitive  skills  for  isolating  malfunctions  or  re¬ 
pairing  equipment.  What  do  we  know  about  the  level  of  fidelity  required  to  ef¬ 
fectively  train  maintenance  tasks  when  they  involve  behavior  more  complicated 
than  switch- pres sing  and  display  reading. 


Perceptual -Motor  Maintenance  Tasks 

In  the  maintenance  environment,  perceptual -motor  skills  are  needed  in  adjustment, 
alignment,  calibration,  disassembly,  or  assembly.  Intuitively,  perceptual- 
motor  tasks  are  those  requiring  "hands-on"  experience  for  effective  training. 
Therefore,  the  principle  guiding  selection  of  simulator  configuration  has 
been  to  provide  the  highest  fidelity  simulation  achievable  or  the  actual  equip¬ 
ment  in  the  event  that  technical  or  cost  factors  limit  the  achievable  fidelity. 
This  has  been  especially  evident  in  the  flight  simulator  area.  Our  literature 
review  did  not  discover  any  empirical  studies  of  del iberately  reduced  fidelity 
for  training  perceptual -motor  tasks  in  the  maintenance  domain.  This  gap  in  the 
literature  would  seem  to  be  an  ideal  one  to  fill.  Chapter  III  of  this  report 
presents  a  paradigm  for  carrying  out  some  necessary  research. 
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Cognitive  Maintenance  Tasks 

The  need  for  the  equipment  maintainer  to  think  rather  than  follow  or  recall  pro¬ 
cedures  can  arise  under  a  number  of  different  circumstances. 

•  The  observable  symptoms  of  a  rtialfunction  must  be  recognized  and 
interpreteo  in  order  to  determine  the  applicable  fault 
isolation  procedure. 

•  Procedural  steps  often  involve  calculation  or  waveform 
interpretation. 

e  Inevitably  procedures  or  automatic  and  built-in  tests  fail  to 
always  isolate  the  specific  source  of  a  system  malfunction. 

This  may  be  because  procedures  have  been  inadequately  docu¬ 
mented  or  because  they  isolate  only  to  a  (large)  group  of 
components.  In  the  case  of  automatic  tests  it  may  be  because 
they  do  not  (or  cannot)  consider  multiple  or  intermittent  causes 
for  a  malfunction. 

Under  any  of  these  circumstances  the  maintainer  becomes  a  problem-solver  or 
decision-maker.  The  observable  symptoms  of  a  malfunction  are  interpreted  in 
light  of  what  the  technician  knows  about  how  the  system  is  configured  and 
how  it  works.  In  non-procedural  troubleshooting  the  maintainer  must  decide 
what  information,  e.g.,  line  voltage,  absence  of  pressure,  etc.,  is  needed  to 
eliminate  or  confirm  suspect  system  components  or  subsystems.  That  is,  he  or 
she  must  in  effect  create  a  troubleshooting  procedure. 

According  to  a  recent  analysis  by  Bond  and  Towne  (1979)  the  main  difficulty  in 
troubleshooting  "is  that  the  technician's  cognitive  map  of  essential  physical 
relations  (electronic,  hydraulic,  electro-mechanical,  and  so  on)  in  a  complex 
equipment  is  often  incomplete,  vague,  or  incorrect"  (pp.  5-6).  We  are  in  agree¬ 
ment  with  this  view,  but  would  extend  it  to  include  the  technician's  usually 
faulty  mental  inode 1 1  of  how  the  system  functions  including  the  functional  re¬ 
lationships  among  subsystems.  These  problems  are  compounded  by  the  use  of 


*0ne  can  think  of  a  mental  model  as  a  simulation  residing  in  the  technician's 
head.  He  can  "run"  the  simulation  to  discover  what  might  happen  under  certain 
circumstances. 
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suboptima'I  troubleshooting  strategies  even  by  relatively  experienced  trainees 

(Hunt  and  Rouse,  1981).  For  example,  although  the  half-split  troubleshooting 
•  2 

technique  is  taught  in  technical  schools,  it  is  not  consistently  used  on  the 
job. 

We  know  surprisingly  little  about  the  relationship  between  device  fidelity  and 
training  effectiveness  for  cognitive  troubleshooting  skills.  Many  trainers 
have  been  built  over  the  last  15-20  years  (Valverde,  1968  and  Fink  and  Shriver, 

1978  present  lengthy  reviews).  These  include  flat-panel,  computer  graphics 
terminal  and  three-dimensional  devices.  Furthermore,  computer  simulation  models 
have  been  developed  to  train  the  mental  model  aspects  of  electronic  circuit 
troubleshooting  (Brown,  et  al.,  1975;  and  Brown  and  Burton,  19/5),  and  gen¬ 
eral  and  system  specific  troubleshooting  strategies  (May,  et  al.,  1976; 
brooks,  et  al.,  1377;  Rouse,  1979  a,  b;  and  Hunt  and  Rouse,  1381).  The  common 
thread  tying  all  of  these  devices  and  simulations  together  is  the  relatively 
high  degree  of  functional  similarity  present  across  varying  levels  of  physical 
similarity. 

In  no  instance  has  more  than  one  type  of  device  or  simulation  been  compared  to 
the  same  actual  equipment  for  assessing  training  effectiveness.  The  Air  Force 
Human  Resources  Laboratory,  however,  is  currently  conducting  such  an  evaluation 

(Cicchinel 1 i ,  et  al.,  1980).  In  general,  the  results  of  most  "one  on  one"  comparisons 
indicate  that  the  training  device  is  equally  effective  as  the  actual  equipment 
for  training  (Fink  and  Shriver,  1978;  Orlansky  and  String,  personal  conmuni cation) . 

These  studies,  however,  are  plagued  by  a  number  of  weaknesses,  some  of  which  are 
enumerated  by  Cicchinelli  (1980).  Paramount  among  these  is  the  inability  to 
collect  true  transfer  of  training  data  because  the  actual  equipment  cannot  be 
faulted.  Most  researchers  (c.f..  Miller  and  Rockway,  1975;  Rigney,  et  al., 

1978  a,  b)  are  therefore  limited  to  assessing  device  training  effectiveness  in 

•x— ——————— 

In  the  half-split  technique  one  attempts  to  successively  reduce  the  size  of  the 
set  of  possible  faulty  subsystems  or  components  in  half  through  each  measurement. 
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in  terms  of  troubleshooting  performance  on  the  device  itself. 

We  thus  lack  systematic  and  valid  data  on  the  device  fidelity  required  for 
effective  training  of  the  various  aspects  of  cognitive  troubleshooting  skills. 
Chapter  III  presents  a  preliminary  research  plan  designed  to  provide  the  needed 
data. 

FIDELITY  RESEARCH  AND  SUBJECT  POPULATIONS 

The  target  population  (i.e.,  the  one  to  which  we  seek  to  generalize  our  results) 
for  the  research  effort  is  first-term,  U.S.  Army  enlistees  selected  for  maintenance 
job  technical  training.  We  will  not  be  able  to  conduct  our  research  using  Army 
personnel.  College  sophomores,  for  example,  may  not  represent  a  comparable  group. 

In  order  to  insure  full  general izability  of  the  research  results,  we  must  select 
subjects  as  consistently  as  possible  according  to  characteristics  of  the  target 
population.  These  characteristics  include  demographic  and  individual  difference 
variables.  For  example,  ideally  the  subjects  should  have  the  same  average  edu¬ 
cation  level  and  mechanical  or  electronic  aptitude  as  Army  maintenance  trainees. 

In  practice  it  will  be  impossible  to  match  the  subject  and  target  populations  on 
all  variables  or  even  every  important  variable.  Rather  we  will  have  to  prioritize 
the  important  variables  and  choose  a  subject  population  on  this  basis. 

The  research  descirbed  in  this  plan  is  designed  to  explore  general  relationships 
between  simulator  fidelity  and  training  effectiveness.  It  will  be  the  job  of 
future  research  efforts  to  examine  the  effects  of  subject  characteristics  on 
this  general  relationship. 

SUMMARY 

In  this  chapter  we  have  developed  a  fidelity  research  framework  through  con¬ 
siderations  of  1)  operational  definitions  of  fidelity,  2)  economic  sequencing 
of  studies,  3)  theoretical  and  empirical  aspects  of  maintenance  tasks  and  re- 
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quired  skills,  and  4)  desired  subject  population  characteristics.  The  research 
strategy  and  guidelines  established  will  be  instantiated  in  the  research  plan 
presented  in  Chapter  III. 
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CHAPTER  III 


PILOT  STUDY  DESIGN  CONSIDERATIONS  AND  PARADIGMS 


In  the  previous  chapter  we  developed  a  research  framework  through  a  consideration 
of  strategy,  tasks,  and  subject  copulation  characteristics.  In  this  chapter 
we  oresent  potential  research  paradigms  for  studying  the  relationship  between 
training  simulator  fidelity  and  transfer  of  training  for  perceptual -motor 
and  cognitive  tasks.  Before  describing  the  research  paradigms,  we  turn  to 
a  discussion  of  general  design  considerations  for  the  pilot  studies. 

GENERAL  DESIGN  CONSIDERATIONS 

There  are  three  guidelines  that  will  govern  the  design  of  our  laboratory-based 
pilot  studies. 

•  Study  part-  or  micro-tasks. 

•  Determine  appropriate  manipulations  of  fidelity. 

•  Utilize  transfer  of  training  paradigm. 

We  next  briefly  discuss  each  of  these  guidelines  before  turning  to  a  descrip¬ 
tion  of  specific  research  paradigms. 

Micro  Tasks 


i he  to-be  learned  tasks  must  meet  certain  criteria.  They  must  be  representa¬ 
tive  of  the  skills  required  in  an  actual  maintenance  task  environment!  task 
oerformance  must  be  easily  measured  and  the  measurements  must  be  valid, 
reliable,  and  sensitive;  and  the  tasks  must  be  leamable  in  a  reasonable  oeriod 
of  time.  One  way  to  meet  these  criteria  is  to  study  parts  of  tasks  or 
micro-tasks  rather  than  whole  tasks.  For  examole,  cognitive  tasks  have 
structural  (i.e.,  mental  model)  and  process  (i.e.,  troubleshooting  strategy) 
components.  Legitimate  research  Questions  can  be  raised  and  answered  about 
each  comoonent. 


Manipulation  of  Fidelity 

As  discussed  earlier  in  this  reoort,  fidelity  is  a  summary  concept  for 
characterizing  the  configuration  and  capabilities  of  a  training  simulator. 

In  the  oroDOsed  research  we  are  primarily  concerned  with  the  effect  of  the 
(physical  and  functional)  similarity  between  devices  and  the  actus  1  equipment 
on  training  effectiveness.  Our  aim  is  to  hold  other  factors  that  may 
impact  effectiveness,  e.g.,  instructional  method,  constant,  or,  in  the  case 
of  device  instructional  features,  null  in  value.  This  is  done  as  a  first 
step.  Future  research  efforts  will  ouild  on  the  framework  established  here 
to  examine  the  interactive  effects  of  instructional  method  and  other  variables 
such  as  task  type,  task  difficulty,  device  acceptance,  and  many  others. 

There  are  many  choices  to  be  made  in  the  manipulation  of  fidelity.  An  over¬ 
riding  concern  is  that  we  define  experimental  conditions  that  differ 
significantly  along  defined  dimensions  of  fidelity. 

We  have  therefore  defined  a  number  of  criteria  or  constraints  that  our  manipu¬ 
lations  of  fidelity  must  meet.  First,  the  mani uolations  must  be  consistent 
with  the  theoretical  distinction  between  ohysical  and  functional  similarity. 
Initially,  both  physical  and  functional  similarity  must  bear  the  same  general 
relation,  i.e.,  high,  medium,  or  low,  to  the  actual  equipment. 

Second,  the  alternative  configurations  must  be  of  interest  to  the  Army  for 
reasons  of  either  cost  or  resolving  decisons  among  competing  technologies 
having  a  DOtential  training  application. 

Third,  the  properties  of  the  task  that  require  learning  via  practice  must  be 
understood,  the  more  thoroughly,  the  better.  In  this  manner  we  can  con¬ 
fidently  manipulate  the  fidelity  of  those  aspects  of  the  actual  equipment 
that  are  operative  in  skill  acquisition. 

Fourth,  by  the  same  token,  the  properties  of  the  physical  system  embodied 
by  the  actual  equipment  must  be  understood  so  that  functional  characteris¬ 
tics  can  be  specified  and  precisely  manipulated. 
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Fifth,  any  alternative  configurations  chosen  must  be  technologically  and 
economically  feasible. 


Sixth,  and  finally,  the  manipulation  of  fidelity  must  be  indeoendently  assessed 
through  a  consensus  of  observer  judgment  and  ratings.  We  must  have  confidence 
that  our  low  fidelity  condition,  for  examole,  actually  is  low  in  fidelity. 

Transfer  of  Training  Paradigm 

We  are  most  interested  in  how  various  alternative  device  configurations  differ 
with  respect  to  conventional  training,  i.e.,  training  with  actual  eauipment. 

The  key  issue  is  to  determine  whether  or  how  much  training  time  can  be  saved 
by  substituting  a  training  device  for  the  actual  equipment.  Criterion 
measurement  must  be  on  the  actual  equionent.  The  transfer  of  training 
paradigm  is  thus  appropriate. 

Each  experiment  will  consist  of  an  acquisition  phase  and  a  transfer  Dhese. 
Acquisition  will  take  place  in  the  context  of  the  actual  equipment  (control) 
or  alternative  device  configurations.  Transfer  in  terms  of  savings  will  be 
measured  on  the  actual  equipment.  At  later  stages  of  the  research  the 
transfer  conditions  may  be  manioulated  to  reveal  not  only  how  much  and 
what  has  been  learned,  but  also  how  flexible  or  stable  the  learning  is.  For 
examole,  task  difficulty  could  be  operational ized  as  problem  complexity  or 
time  stress.  Time  pressure  would  be  an  appropriate  difficulty  factor  to 
manipulate  because  military  skills  ultimately  must  transfer  to  a  war  time 
context  in  which,  e.g.,  time  to  repair  equipment  is  critical. 

RESEARCH  PARADIGMS 

The  discussion  in  this  section  is  organized  around  task  type. 

Perceptual -Motor  Task  Pilot  Research 

The  goal  of  this  line  of  research  is  to  determine  the  effect  of  physical 
and  functional  training  simulator  similarity  on  transfer  of  training  for 
perceptual -motor  tasks.  In  the  maintenance  task  environment,  perceptual- 
motor  skills  are  required  in  adjustment,  alignment,  assembly,  disassembly. 


and  so  on.  Maintenance  of  mechanical  systems,  e.g.,  vehicles,  accounts  for 
a  major  Army  training  requirement.  Training  under  conventional  conditions, 
using  actual  equipment,  is  costly  and  can  be  hazardous  to  equipment  or 
personnel.  Alternative  training  conditions  are  thus  desirable. 

For  the  purposes  of  this  research  we  need  a  task  that  can  be  studied 
effectively  in  the  laboratory,  where  the  required  performance  is  representa¬ 
tive  of  Army  maintenance  tasks.  We  have  selected  the  trueing  of  a 
bicycle  wheel  because  it  appears  to  meet  these  needs. 

The  wheel  is  a  simple  yet  elegant  physical  system  for  translating  power  into 
motion.  The  geometry  of  a  wheel  and  the  relative  rigidity  of  its  surface 
and  the  surface  on  which  it  moves  determine  the  efficiency  of  the  power 
transformation. 

Bicycle  wheels  should  minimize  weight.  SuoDort  for  the  lightweight  metal, 
semi-rigid  rim  is  provided  by  (adjustable)  sookes.  In  use,  the  geometry 
(i.e.,  shape)  of  this  type  of  wheel  changes,  causing  it  to  become 
imperfect.  Because  required  power  for  motion  increases  with  deviation  from 
true,  a  great  premium  is  placed  on  keening  these  wheels  true. 

Trueing  a  wheel  is  not  a  simple  matter.  The  task  is  complex  enough  to  be 
frustrating  to  a  novice,  yet  appears  to  be  mastered  in  a  reasonable  amount 
of  time--after  trueing  5-10  wheels  according  to  expert  opinion. 

The  task  consists  of  first  detecting  any  misalianment  (e.g.,  correctly 
attributing  wobble  to  the  wheel  and  not  a  loose  axle),  its  locations )  and 
amount,  and  then  manipulating  the  spoke  nipples  (see  Figure  2  )  with  a 
spoke  wrench  to  correct  it.  Figure  2  shows  the  typical  configuration  of 
the  spokes  attached  to  a  rim. 

Misalignment  is  detected  by  spinning  the  wheel  in  the  context  of  fixed  reference 
points  on  either  side  of  the  rim  (e.g.,  the  brake  pads  if  the  wheel  is  on  the 
bike).  The  principle  involved  in  correcting  the  deviation  is  to  loosen, via 
the  spoke  nipples, the  spokes  that  go  to  the  side  of  the  hub  that  the  rim 
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Figure  2.  Typical  Configuration  of  a  Bicycle 
Wheel.  (After  Garvey,  1972). 
Illustrated  are  the  spoke  wrench 
and  the  principle  involved  in 
straightening  the  rim  (see  text). 


pulls  toward  and  tighten  the  spokes  to  the  other  side.  This  is  a  Drecision 
operation  involving  increasingly  smaller  adjustments  of  spokes  farther  away 
from  the  point  of  maximum  deviation.  The  studies  described  next  are  aimed 
at  discovering  the  effects  of  manipulating  the  fidelity  of  the  training 
medium  on  the  training  transfer  of  a  perceptual -motor  skill. 

Perceptual -Motor  Pilot  I --The  Duroose  of  this  pilot  study  is  to  establish 
an  expert  baseline  for  the  task.  The  temporal  and  performance  measurement 
characteristics  of  the  task  will  be  determined.  Several  experienced 
individuals  will  participate  as  subjects.  They  will  be  required  to  true 
a  wheel  of  known  but  variable  deviation  from  true.  A  wheel  stand  will  be 
instrumented  for  measurement  using  a  travel  dial  indicator.  The  indicator 
measures  degrees  of  displacement  from  a  fixed  reference.1 

The  results  of  this  study  will  provide  a  characterization  of  the  performance 
of  a  trained  individual.  Of  particular  interest  are  the  length  of  time  it 
takes  to  true  the  wheel  under  different  amounts  of  deviation  (difficulty), 
and  whether  the  measurement  technique  described  above  needs  to  be  supplemented 
by.  e.g.,  observation  of  tool  use. 

Perceptual -Motor  Pilot  I I --The  Duroose  of  this  study  is  to  establish  the 
parameters  of  performance  acquisition.  The  temporal  and  performance  charac¬ 
teristics  of  task  acquisition  will  be  determined  and  a  training  method  will 
be  developed.  Several  naive  individuals  will  participate  as  subjects.  They 
will  be  required  to  learn  how  to  true  the  wheel  under  conventional 
training  conditions.  Particular  attention  will  be  given  to  the  method  used 
for  training  and  difficulty  of  the  task  under  acquisition  and  transfer. 

The  experimenter  (E)  will  be  experienced  in  the  task  and  role  play  as  an 
instructor  during  the  acquisition  phase.  (It  may  prove  difficult  for  £  to 
perform  both  functions,  in  which  case  we  will  need  a  second  individual  to 
help  run  the  study.)  As  the  instructor,  E  will  explain  the  principle  of 


Note  that  spoke  tension  is  also  considered  an  imoortant  measure;  it  should 
be  eaually  distributed.  It  is  possible,  but  time  consuming  to  measure  spoke 
tension. 
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wheel  trueing  and  demonstrate  detection  of  wobble  and  use  of  the  tool.  The 
S  will  practice  while  answers  questions  and  provides  feedback.  This  process 
will  continue  until  a  pre-established  criterion  (based  upon  the  results  of 
study  I)  is  attained. 

The  results  of  this  study  will  allow  us  to  estimate  the  length  of  time  needed 
for  task  acquisition.  This  data  is  critical  for  detailed  planning  of  the 
remaining  studies.  Also,  the  data  will  indicate  the  shape  of  the  learning 
curve  for  the  task  and  provide  an  estimate  of  performance  variability. 

These  data  will  be  critical  to  determining  an  appropriate  level  of  difficulty 
of  the  task  for  training  and  transfer,  and  to  estimating  required  sample  size 
for  the  remaining  studies.  Finally,  during  the  study,  we  will  perfect  and 
standardize  the  training  methodology. 

Perceptual -Motor  Pilot  III--The  purpose  of  this  study  is  to  establish  the 
general  effect  of  fidelity  on  task  training.  Two  simulated  wheels  will  be 
evaluated  as  training  (practice)  media  for  task  acquisition.  One  of 
the  simulated  wheels  will  be  of  medium  fidelity  and  the  other  of  low  fidelity. 
Degradation  of  physical  and  functional  similarity  will  be  matched  as  closely 
as  possible.  Level  of  fidelity  will  be  independently  assessed  through  a 
magnitude  comparison  in  which  ratings  Of  fidelity  (in  terms  of  physical 
and  functional  similarity)  will  be  obtained  from  experienced  individuals 
(subject  matter  experts). 

The  subjects  in  this  study  will  be  naive  to  the  wheel  trueing  task.  The 
training  method  will  involve  demonstration/observation  and  practice  with 
feedback  as  described  in  study  II.  It  is  of  utmost  importance  that  the 
same  method  be  used  for  training  with  both  simulated  wheels.  Task  difficulty 
levels  during  practice  will  be  matched  as  closely  as  possible  to  one 
another  and  to  that  shown  appropriate  in  study  II.  Task  difficulty  during 
transfer  will  be  chosen  so  as  to  maximize  the  chance  of  finding  differences 
between  the  alternative  training  devices. 

The  results  of  this  study  will  establish  whether  we  have  an  effective,  i.e., 
significant,  manipulation  of  fidelity.  The  data  of  this  study  can  be 
analyzed  in  t*e  context  of  data  from  study  II,  in  order  to  provide  an 
indication  of  the  overall  magnitude  of  the  effect. 
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The  nature  of  the  pilot  studies  following  this  experiment  will  deoend  upon 
the  results  of  Study  III.  Assuming  positive  results.  Study  IV  will  be  as 
described  next.  However,  negative  results  (lack  of  a  fidelity  effect)  will 
cause  us  first  to  reexamine  the  fidelity  manipulation,  and  the  difficulty 
of  the  transfer  task.  Next  we  will  change  our  selection  of  subjects  who 
ud  until  this  point  have  been  easily  accessible  (i.e.,  Honeywell  emoloyees) 
and  unpaid  for  their  services. 

Perceptual -Motor  Pilot  IV--The  purpose  of  this  study  is  to  replicate  a  portion 
of  our  orevious  results  in  the  target  population  of  interest.  Pilot  Study  II 
will  be  repeated  with  refined  training  method  and  appropriate  levels  of  dif¬ 
ficulty.  However,  the  subjects  in  this  study  will  be  naive  individuals  chosen 
from  volunteer  high  school  students.  These  volunterrs  will  be  recruited  from 
the  population  of  students  that  will  likely  not  continue  on  in  their  education. 

The  characteristics  of  this  population  can  be  established  through  demographic 
means.  Liaison  with  a  local  high  school  will  be  established  and  within  the 
limits  of  propriety  and  right  to  privacy,  we  will  work  with  school  officials 
to  pinpoint  the  appropriate  pool  of  students.  If  aptitude  test  scores  are 
available  and  accessible,  we  will  use  them  as  a  basis  for  identifying  the 
population  of  interest.  In  this  manner,  a  more  representative,  homogeneous 
group  of  subjects  can  be  isolated.  These  subjects  will  be  paid. 

The  reason  for  not  using  this  population  earlier  is  chiefly  economic.  A 
delay  'ill  also  allow  time  for  various  administrative  procedures  and  approvals 
necessary  to  gain  access  to  the  population  of  interest.  For  examnle,  ARI 
and  Honeywell  must  insure  that  the  research  methods  and  procedures  are  con¬ 
sistent  with  Government  regulations  for  the  use  of  human  subjects. 

Assuming  positive  results  of  this  study,  we  will  be  in  a  position  to  collect 
data  on  the  training  effectiveness  of  alternative  device  configurations. 

Perceptual -Motor  Pilot  V--This  study  will  replicate  Study  III  in  which  low  and 
medium  fidelity  devices  were  used  for  training.  High  school  students  in  a  non¬ 
college  track  will  serve  as  subjects.  At  this  stage  no  finer  categorization 
of  subjects  will  be  made. 
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Perceptual -Motor  Pilot  VI— The  purpose  of  this  study  will  be  to  determine  if 
training  effectiveness  for  this  task  is  a  differential  function  of  physical 
or  functional  similarity  of  the  simulated  wheel(s)  to  the  actual  wheel.  We 
will  devise  two  additional  devices,  one  of  which  will  have  high  physical,  but 
low  functional  similarity,  while  the  other  will  be  of  low  physical  but  high 
functional  similarity.  As  in  Study  III  we  will  obtain  an  independent  assess¬ 
ment  of  the  degree  of  similarity.  In  all  other  respects  this  study  will  be  a 
replication  of  Studies  III  and  V.  At  this  stage  in  the  research  we  should  be 
able  to  use  high  school  students  as  subjects.  The  results  of  this  study  will 
indicate  the  relative  effects  of  both  aspects  of  fidelity  as  determiners 
of  training  effecti veness  for  perceptual -motor  tasks.  Subsequent  research 
can  fill  in  additional  levels/combinations  of  physical  and  functional 
similarity. 

Cognitive  Task  Pilot  Research 

The  ultimate  goal  of  this  line  of  research  is  to  determine  the  effects  of  vary¬ 
ing  levels  of  fidelity  on  the  effectiveness  of  training  troubleshooting  skills. 
At  the  outset,  however,  the  research  effort  must  be  directed  at  developing 
methodological  and  procedural  tools  for  studying  cognitive  task  acquisition  and 
performance. 

We  have  contended  elsewhere  in  this  report  that  cognitive  task  performance 
in  equipment  troubleshooting  is  based  on  (a)  a  cognitive  model  or  representa¬ 
tion  of  the  system  in  question,  and  (b)  a  strategy  for  problem-solving. 

There  are  a  large  number  of  questions  to  be  asked  about  both  the  structure 
(model)  and  the  process  (strategy).  For  example, 

•  How  do  we  know  if  a  model  or  strategy  has  been  acquired?  Is  being 
used? 

•  How  does  the  quality  of  the  model  or  strategy  vary  with  training 
conditions,  e.g.,  fidelity? 

•  How  do  models  or  strategies  vary  with  the  nature  of  the  system? 

The  skill  level  of  the  user? 
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•  Are  some  tasks  better  taught  using  models  than  others? 

•  How  do  we  choose  the  appropriate  model  or  strategy  to  teach? 

•  Are  some  strategies  suitable  to  some  models,  but  not  others? 


Prior  to  being  able  to  carry  out  a  series  of  studies  as  outlined  in  the 
previous  section,  we  must  develop  a  method  for  defining  cognitive  models  and 
strategies  and  measuring  the  quality  of  decision-making  in  a  cognitive  task. 
Equally  importantly,  we  must  develop  or  select  a  system  and  design  a  trouble¬ 
shooting  task  suitable  for  laboratory  study.  In  the  near-term,  we  will 
proceed  along  two  parallel  paths  directed  at  the  methodological  questions  and 
the  system/task  questions. 

Measurement  Methodology  Development--The  purpose  of  this  pilot  study  is  to 
develop  a  method  for  assessing  the  quality  of  the  models  and  strategies 
underlying  performance  in  a  complex,  conceptual  task.  The  method  must  provide 
data  allowing  inferences  about  internal  structures  and  covert  strategies 
and  permitting  discrimination  among  structures  and  strategies  of  different 
quality. 

An  analysis  of  troubleshooting  cannot  consist  of  a  listing  of  procedures 
followed,  since  the  skills  of  maintenance  troubleshooting  may  not  consist 
of  following  procedures.  Skilled  technicians  appear  to  be  using  task  and 
contextual  information  to  make  decisions  and  accomplish  a  variety  of  long- 
range  and  short-range  goals.  Accordingly,  our  methods  will  focus  on  the 
way  skilled  personnel  use  available  information,  and  on  the  types  of  goals 
they  are  concerned  with.  If  the  methods  evaluated  are  successful  in 
accomplishing  these  objectives,  we  should  be  able  to  use  them  not  only  to 
make  inferences  about  the  quality  of  underlying  models  and  strategies,  but 
to  provide  guidance  about  the  types  of  goals  trainees  must  learn  to  be 
aware  of  and  to  use;  the  methods  should  also  result  in  guidance  about  the 
type  of  decision  options  that  must  be  represented  by  the  training  device  in 
order  for  the  trainee  to  learn  to  use  goals  to  select  between  critical 
alternative  pathways. 
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Therefore,  the  desired  output  for  this  line  of  research  is  a  generalizable 
method  that  can  ultimately  be  used  to  tell  training  device  designers  some 
specific  requirements,  in  terms  of  goals  represented  and  decision  options 
presented.  This  guidance  should  constitute  a  partial  resolution  for  questions 
of  fidelity. 


Rationale- -The  Daradigm  that  will  be  assessed  is  a  synthesis  of  work  by 
Attneave  (1954)  and  Shannon  (1951)  on  information  theory,  with  work  by  Klein 
and  Klein  (1981)  studying  goal  networks  of  proficient  CPR  paramedics. 

Attneave  (1954)  discussed  a  paradigm  in  which  a  picture  is  constructed  of 
three  objects,  each  with  a  different  color,  and  divided  into  a  matrix  of 
50  rows  and  80  columns,  to  produce  4,000  cells.  The  subject  never  sees  this 
picture.  The  task  is  to  guess  the  color  of  each  cell.  They  predict  one  cell, 

are  given  feedback  about  the  actual  color,  and  move  on  to  the  next  cell. 

Since  only  three  colors  are  used,  the  probability  of  guessing  correctly  by 
chance  is  approximately  1/3  for  each  cell.  Actually,  subjects  should  only 
make  15-20  errors  in  the  whole  matrix,  because  they  can  use  the  redundancy 
of  the  picture  (shapes,  contours,  symmetries).  This  paradigm  was  suggested 
as  a  way  of  exploring  the  information  value  of  a  visual  figure,  but  it  can 
also  be  used  to  probe  a  subject's  ability  to  recognize  and  use  redundancies. 
That  is,  if  a  complex  domain  is  used,  subjects  with  more  expertise  will  be 
expected  to  make  more  accurate  predictions  about  elements  than  subjects 
with  little  or  no  experience. 

This  method  can  only  be  used  for  tasks  that  can  be  analyzed  into  elements, 
but  this  would  include  tasks  such  as  troubleshooting  of  equipment,  where  there 
is  a  sequence  of  tests  and  results.  If  the  methods  are  fruitful,  during 
Phase  II  of  SIMTRAIN,  we  could  analyze  a  troubleshooting  problem  into  the 
tests  and  results  sequentially  obtained  during  the  course  of  the  diagnosis. 

The  same  problem  would  then  be  presented  to  a  different  technician,  whose 
task  would  be  predicting  which  test  would  be  made  next,  followed  by  feed¬ 
back  on t he  results  of  that  test,  followed  by  a  request  to  guess  the  next 
test  performed. 
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This  paradigm  can  tell  us  about  the  level  of  redundancy  in  the  task,  but  that 
is  not  enough  to  understand  training  requirements.  We  need  to  understand  the 
choices  being  considered.  This  is  accomplished  by  asking  the  subject  which 
are  the  likely  alternatives  at  each  choice  point.  By  testing  subjects  at 
different  levels  of  proficiency,  we  can  identify  changes  in  the  types  of 
alternatives  they  are  considering  to  perform  a  task.  Klein  and  Klein's  (1981) 
CPU  work  has  shown  that  more  experienced  personnel  can  be  shown  to  utilize 
goal  dimensions  that  less  experienced  personnel  cannot 

With  regard  to  the  question  of  fidelity,  we  hope  to  be  able  to  ultimately  use 
this  paradigm  to  answer  design  questions  about  the  number  and  types  of  alterna¬ 
tives  that  must  be  presented  at  choice  points,  for  training  personnel  at  any 
given  level  of  proficiency. 

The  specific  pilot  study  to  be  performed  will  evaluate  the  paradigm  within  the 
domain  of  chess-playing  expertise.  The  game  of  chess  was  selected  because  it 
is  a  decision  task  that  can  be  analyzed  into  elements.  Subjects  can  be  easily 
assigned  to  different  skill  levels,  and  t'  ere  is  good  availability  of  motivated 
subjects.  Finally,  there  are  no  equipment  costs.  All  of  these  factors  will 
allow  us  to  perform  the  pilot  study  and  ana^ze  the  results  within  available 
resources 

Design--We  will  be  contrasting  three  groups--experienced  chess  players 
(rated  1800-2100),  average  players  (rated  1500-1600),  and  beginners  (rated 
up  to  1400).  At  least  ten  subjects  will  be  run  in  each  group.  The  materials 
will  consist  of  three  chess  games  actually  played  by  players  rated  as  experts 
(2000-2200).  Each  subject  will  attempt  to  predict  the  moves  played  for  each 
of  the  games.  For  each  game,  the  subject  will  be  shown  the  chess  board  after 
move  10.  l hen,  the  subject  will  be  asked  which  move  was  made  by  white  on 
move  11.  ihe  subject  will  list  the  likely  alternatives,  as  well  as  the  actual 
guess,  ihe  subject  will  then  be  told  if  the  guess  was  correct  or,  if  not, 
which  move  was  actually  played.  This  process  will  continue  for  a  sequence 
of  20  moves. 
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I  he  data  will  be  analyzed  in  several  ways.  First,  we  will  compute  accuracy  of 
guesses  by  calculating  the  number  correct  out  of  the  20  moves  studied.  Second, 
we  will  compute  the  accuracy  of  subjects'  ability  to  identify  plausible  options, 
by  calculating  the  number  of  accurate  alternatives  listed,  and  dividing  this 
by  the  total  number  of  alternatives  listed.  This  will  enable  us  to  see  if 
accuracy  of  guessing  increases  with  skill  level,  along  with  ability  to 
recognize  plausible  options.  By  studying  the  patterns  of  novice  and  experi¬ 
enced  chess  players  in  generating  options,  we  should  be  able  to  learn  more 
about  how  this  type  of  decision  making  task  is  performed. 

Successful  completion  of  this  study  and  the  analyses  described  will  provide 
methodological  tools  useful  for  two  purposes.  First,  using  the  prediction 
technique  we  should  be  able  to  derive  implications  about  the  simulation 
requirements  for  ensuring  that  an  adequate  range  of  alternatives  is  represented 
by  a  device. 

Second,  it  should  be  possible  to  use  these  methods  to  evaluate  the  effective¬ 
ness  of  training  devices  and  programs.  Personnel  at  higher  skill  levels 
should  show  higher  predictive  accuracy.  Once  validated  for  chess  we  can 
apply  the  methods  to  cognitive  troubleshooting  tasks.  The  technique  may  be 
of  particular  value  in  tasks  where  there  is  no  "right"  answer;  a  measure  of 
predictive  accuracy  will  be  a  useful  means  of  evaluating  performance  in 
non-procedural  troubleshooting. 

System  Selection  and  Task  Design 


The  purpose  of  this  line  of  research  is  to  assess  the  properties  of  different 
systems  and  tasks  in  light  of  the  requirements  of  laboratory  research. 

These  requirements  and  some  consideration  of  options  for  meeting  them  are 
briefly  summarized  below. 

The  system  must  be  related  in  an  obvious  or,  if  possible,  exact  way  to 
an  actual  Army  system.  Troubleshooting  must  be  consistent  with  the 
maintainence  philosophy  of  such  a  system.  In  short,  both  the  system  and 
task  must  have  face  validity  and  minimize  artif icialty. 
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The  system  must  provide  for  a  cognitively  challenging  task.  Therefore, 
it  must  be  relatively  complex,  consisting  of  subsystems  built  up  of 
components,  as  opposed  to  a  simple  system  consiting  only  of  components. 

A  complex  system  will  provide  the  required  task  ambiguity  and  also 
the  capability  to  vary  difficulty  —  troubleshooting  can  be  accomplished 
at  either  or  both  the  system  and  subsystem  levels. 

The  system  must  lend  itself  readily  to  simulation.  A  system  with  digital 
signals  would  require  less  effort  in  this  regard  than  one  with  analogue 
signals.  However,  an  analogue  system  is  more  conceptually  complex  and 
may  provide  better  discrimination  of  fidelity  effects. 

Learning  the  task  cannot  require  lengthy  participation  on  the  part  of 
the  subject,  Therefore,  the  troubleshooting  task  should  be  trainable 
with  either  minimal  pre-training  or  pre-training  associated  with  cotrmon 
courses  of  vocational  technical  study  (e.g.,  basic  electronics)  so 
that  a  subject  pool  would  be  readily  available. 

Finally,  the  system  and  task  should  be  accessible  for  replication  in 
other  laboratories.  Therefore,  nothing  about  either  should  be  especially 
unique  or  require  esoteric  equipment,  programming,  skills  or  knowledge. 

There  are  two  candidate  systems  under  consideration  at  present.  One  is  a 
radio  receiver,  the  other  a  microcomputer  system. 

In  the  course  of  this  research  we  will  explore  the  suitability  of  both  by 
having  one  or  two  individuals  become  expert  in  their  repair.  These 
individuals  will  use  a  variety  of  techniques  including  interviews  with 
experts  and  hands-on  experience.  The  decision  making  method  described  in  the 
previous  subsection  might  also  be  used  with  experts. 

SUMMARY 

In  this  chapter  we  developed  specific  study  designs  for  researching  the 
relationships  between  training  simulator  fidelity  and  transfer  of  training. 
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for  perceotual  motor  and  cognitive  maintenance  tasks.  The  prooosed 
experimental  methods  were  based  uDon  the  need  to  (1)  study  part-  or 
micro-tasks;  (2)  determine  aooroDriate  manipulations  of  fidelity:  and 
(3)  utilize  a  transfer  of  training  Daradigm.  The  perceptual -motor 
fidelity  research  will  utilize  a  bicycle  wheel  trueing  task;  a  series  of 
six  studies  were  described.  The  cognitive  fidelity  research  requires  the 
develoDment  of  measurement  method  and  selection  of  a  specific  experimental 
task;  the  necessary  research  and  analysis  was  described. 
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OF  THE  ARMY'S  TRAINING 
SIMULATOR  DEVELOPMENT  PROCESS 


HBCSONQ  PAOX  BUMC-NOT 


59 


ARMY  TRAINING  SIMULATOR  DEVELOPMENT 


The  following  description  of  the  Army's  experimental  training  simulator 
development  process  is  based  upon  the  detailed  artline  found  in  TRADOC 
Circular  70-80-1  and  the  survey  data  collected  during  our  site  visits. 

We  describe  the  process  herein  so  as  to  identify  points  that  impact  device 
configuration  and  capabilities.  In  this  manner  key  leverage  points  for 
furnishing  guidance  can  be  isolated.  These  points  are  identified  in  a 
subsequent  discussion  of  some  of  the  strengths  and  weaknesses  of  the  de¬ 
sign  process.  A  study  approach  for  resolving  some  of  the  identified 
weaknesses  is  described. 

There  are  two  different  sequences  of  action  that  may  be  pursued  in  the  train¬ 
ing  device  development.  The  one  used  depends  upon  whether  the  device  will 
support  a  developing  weapon  system  or  whether  it  will  support  one  or  more 
fielded  systems.  It  is  usually  in  this  latter  instance,  called  nonsystem 
training  device  development,  that  PM  TRADE  becomes  involved.  However,  at 
the  request  of  the  System  Program  Manager,  PM  TRADE  can  become  involved  in 
system  training  device  development.  Figure  A-l  shows  a  highly  simplified 
version  of  the  nonsystem  development  process.  The  following  description 
applies  specifically  to  this  example.  Aspects  of  nonsystem  and  system  train¬ 
ing  device  development  that  impact  fidelity  specification  are,  in  general, 
the  same.  The  strengths  and  weaknesses  of  the  processes  in  this  report  are 
thus  highly  similar. 

The  initiating  condition  for  nonsystem  training  device  development  is  the 
identification  of  a  training  problem.  If  a  subsequent  media  analysis 
and  selection  indicates  that  a  device  is  required  to  accomplish  training 
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Figure  A-l.  Simplified  Nonsystem  Training  Device  Development 

(approval  cycles  not  shown;  commerlcal  availability 
not  shown;  see  Appendix  C,  Figure  C-l  of  TRADOC 
Cir  70-80-1  for  more  detail). 


then  a  training  device  need  statement  (TDNS)  is  generated.  The  user 
(proponent  school)  is  responsible  for  media  analysis  and  selection.  The 
TDNS  is  forwarded  to  the  Army  Training  Support  Center  (ATSC)  for  action. 

The  important  parts  of  the  TDNS  from  a  device  characteristics  perspective 
are  1)  a  description  of  the  nature  of  the  training  deficiency  in  terms  of 
the  training  capabilities  required,  2)  an  identification  of  the  tasks 
to  be  trained,  and  3)  a  description  of  the  capabilities  of  the  individuals 
who  will  be  the  subject  of  training  with  the  device. 

Upon  submittal  of  the  TDNS,  ATSC  coordinates  with  PM  TRADE  which  determines 
the  availability  of  existing  technology  (cormercial ly  available)  to  meet 
the  training  need.  If  an  off-the-shelf  solution  or  set  of  components  does 
not  exist,  then  ATSC  tasks  (and  subsequently  assists)  the  proponent  with 
developing  a  training  device  letter  of  agreement  (TDLOA)  for  the  purpose  of 
technology  development.  If  technology  is  available  then  the  proponent 
develops  a  training  device  requirements  (TDR)  document  or  a  training  device 
letter  requirements  (TDLR) .  (See  below.) 


There  is  much  information  contained  in  a  TDLOA  which  will  directly  influence 
device  configuration  and  characteristics.  The  format  of  the  TDLOA  is 
presented  in  Fiqure  A-2.  As  can  be  ascertained  from  the  TDLOA  format, 
a  considerable  amount  of  training  requirements  analysis  must  be  accomplished 
in  order  to  provide  the  required  information. 

Upon  submittal  of  a  partial  TDLOA,  the  proponent  is  authorized  to  establish 
a  joint  working  group  (JUG)  for  the  purpose  of  finishing  the  TDLOA,  and 
developing  and  coordinating  additional  requirements  documents.  The 
Principal  Characteristics  section  is  one  highly  important  aspect  of 
the  requirements  documents  that  is  completed  in  the  forum  of  the  JWG. 
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1 .  Need 


a.  A  brief  description  of  the  training  required  to  include  the  tasks  to 
be  supported,  l  lie  cut  rent  method  mud  to  train  the  tasks,  the  training 
syston(s)  to  be  replaced,  and  the  time  r  rnme  tor  which  the'  new  capability  is 
needed. 

b.  Catalog  of  approved  requi re-.euts  documents  (CARDS)  reference  number 
(to  be  assigned  by  DA  ODCSOPS  upon  final  approval  of  the  document). 

2.  Operational  concept  . 

a.  A  description  of  the  rol,  oi  the  training  device  in  the  training 
concept  and  its  relationship  to  otner  training  systems  and  operational 
hardware. 

b.  The  mission  profile  will  bo  attached  at  Annex  A.  This  annex  should 
describe  how  the  device  will  be  used  during  a  training  day  or  days  to  accom¬ 
plish  training.  It  should  describe  a  measure  of  how  long  the  device  will 
operate  while  be  In,;,  used  to  t  cacti  ouch  task  or  series  of  tasks. 

3.  System  description. 

a.  A  statement  indicating  the  principal  characteristics  expected  to  he 
included  in  the  device,  to  include  how  the  device  will  meet  the  training 
requirement,  where  it  will  he  used,  what  the  device  Iooks  line  and  those  tech¬ 
nological  alternatives  that  have  a  reasonable  chance  of  developmental  success. 
Included,  if  applicable,  must  be  any  physical  constraints  or  health  hazards 
that  will  impact  on  design. 

b.  A  discussion  ot  other  service,  .‘JATo/AlK.A,  or  allied  nation  interest  in 
the  Army  devel opment /procurement .  Include  data  on  other  service  or  allied 
developments  with  a  view  toward  establishing  potential  for  standardization/ 
interc.perabi 1 i ty ,  or  co-production.  Include  data  on  potential  for  procurement 
of  allied  nation  items/svstems. 

4.  Prospective  operational  el tectiveness  and  cost.  A  realistic  quantitative 
estimate  of  the  training  effectiveness  that  will  be  gained  from  the  new  train¬ 
ing  device  when  compared  with  the  training  system  being  replaced.  This  para¬ 
graph  should  include  a  subparagraph  which  identifies  the  estimated  upper  cost 
hand  where  the  training  device  fails  to  be  cost  effective. 


Figure  A-2.  Training  Device  Letter  of  Agreement  (TDLOA) 
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5.  system  development.  This  paragraph  is  divided  into  five  paragraphs.' 
operational,  technical,  logistical,  training  and  manpower.  Each  subparagraph 
describes  the  device  unique  events  which  the  training  developer,  materiel 
developer,  logistician  and  administrator  must  undertake  to  insure  the  de-ice 
is  fielded  as  a  safe  and  complete  system.  Include  manpower  constraints 
related  to  mission  area  or  force  level. 

6.  Schedule  and  milestones.  A  list  of  significant  time  phased  events  which 
will  U-  conducted  as  a  result  of  the  TbLOA.  As  a  minimum  government  accept¬ 
ance  of  the  prototype  and  DT/OT  I,  Validation  IPR/ASARC  II/DSARC  II  will  be 
listee. 


7.  funding.  A  broad  estimate  of  the  advanced  development  (AD),  engineering 
development  (ED),  and  unit  flyaway  costs.  The  AD  and  ED  costs  will  be  broken 
down  by  fiscal  year  and  expressed  in  constant  dollars  provided  by  the  materiel 
developer.  This  paragraph  will  also  identify  the  number  of  prototypes  which 
will  bo  fabricated. 

Anne::  A  -  Operational  Mode  Summary/Mission  Profile.  A  list  of  ways  the  device 
will  be  used.  For  example,  a  device  has  the  capability  of  instructing  six 
hard-to-train  tasks.  The  training  developer  must  outline  for  what  period  of 
time  each  task  is  envisioned  to  be  caught  during  the  life  of  the  device. 

Annex  B  -  Coordination  annex.  A  list  of  organizations  which  provided  comments 
with  full  rationale  for  not  including  LOA  comments,  if  any. 

Annex  C  -  Rationale  annex.  Supports  the  various  characteristics  stated  in  the 
TDLOA. 


Fiqure  A-2.  Training  Device  Letter  of  Agreement  (TDLOA) 
(concluded) 
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The  nonsystem  training  device  development  JWG  is  comprised  of  the  following 
principal  mandatory  members : 

o  Proponent  TRADOC  school  (user/subject  matter  expert) 

o  Army  Training  Support  Center  (ATSC-DST) 

o  DARCOM/PM  TRADE 

As  guidance  to  the  proponent  users,  TRADOC  Cir  70-80-1  lists  11  additional 
commands/activities  that  should  be  considered  for  participation  in  any 
JWG. 

As  noted  above,  the  JWG  completes  the  TDLOA  during  a  series  of  meetings. 
Concurrent  with  this  activity,  a  preliminary  training  development  study 
(TDS)  is  developed.  The  TDS  provides  information  on  the  probable  cost¬ 
and  training-effectiveness  of  the  proposed  device  based  upon  task  and  media 
analyses,  plus  additional  data  resulting  from,  e.g.,  contractor  studies 
or  field  use  of  similar  devices. 

Completion  of  the  TDLOA  and  TDS  initiates  a  series  of  staffing/approval 
actions  which  ultimately  result  in  a  demonstration  and  validation  contract 
that  is  the  responsibility  of  PM  TRADE.  The  contract  effort  results  in  a 
prototype  training  device  that  is  subject  to  developmental  and  operational 
testing  (DT/OTI). 

Upon  completion  of  testing,  or  if  technology  is  available,  the  proponent  user 
and  JWG  must  update  the  TDS  and  develop  a  TDR  document  or  a  TDLR,  depending  upon 
projected  device  cost.  Subsequent  to  the  completion  o+  one  ot  these  documents,  a 

of  staffing/approval  actions  takes  place  resulting  in  a  full-scale  develop¬ 
ment  contract.  If  successful,  as  determined  through  developmental  and 
operational  testing  ( DT/OTI I ) ,  the  production  and  deployment  of  the  device 
will  follow. 


senes 
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The  configuration  and  capabilities  of  the  prototype  and  production  training 
simulator  are  based  upon  a  statement  of  work  including  an  engineering 
specification  prepared  by  PM  TRADE. 

In  sumnary,  there  appear  to  be  three  categories  of  events  in  the  device 
development  process  when  decisions  are  made  regarding  device  configuration/ 
characteristics  (including  fidelity).  These  are: 

o  Media  Analysis/Selection  and  TDNS 

0  TOtOA/TDR/TDRl 

o  Specification  for  RFP 

STRENGTHS  ANO  WEAKNESSES  OF  TRAINING  DEVICE  DEVELOPMENT 

As  can  be  observed  from  the  description  above  and  a  review  of  TRADOC  Circular 
80-70-1,  the  training  device  development  process  is  fairly  systematic  in 
terms  of  the  sequence  of  decisions  and  required  actions.  However,  although 
responsibility  and  authority  are  clearly  defined  in  many  instances,  some 
ambiguities  remain.  The  administrative  mechanics  of  the  process  though  de¬ 
tailed  are  not  entirely  consistent  nor  are  they  fully  consistent  with  the 
Life  Cycle  System  Management  Model. 

The  individuals  who  participate  in  the  process  generally  are  motivated,  ded¬ 
icated,  and  competent  within  their  areas  of  expertise.  Specific  problems  do 
arise  in  the  development  process  associated  with  each  of  the  three  categories 
of  events  listed  above. 

Media  Analysis/Selection  and  TDNS 

Media  analysis  is  a  part  of  the  Instructional  Systems  Development  (ISD) 
process  based  in  part  upon  decisions  made  regarding  learning  objectives  and 
classification  of  objectives  into  categories  of  learning.  (A  recent  report 


by  Reiser  and  his  associates  at  Florida  State  University  (Reiser,  et  al., 
1981)  provides  a  current  example  of  a  media  selection  model.)  If  the  ISO 
process  has  been  used  then  this  requirement  presents  no  problem.  However, 
rarely  is  the  need  for  a  (nonsystems)  training  device  identified  through  a 
systematic  analysis  of  the  tasks  (see  previous  section)  to  be  trained. 
Therefore  the  information  necessary  to  perform  a  media  analysis  is  rarely 
available.  Even  if  this  information  were  available,  the  media  selection 
procedure  is  highly  subjective  and  requires  judgments  that  imply  a  great 
deal  of  knowledge  of  cost  and  feasibility  of  alternatives,  e.g.,  a  training 
device  or  simulator.  The  analysis  and  selection  of  media  requirements  is  a 
difficult  task  to  do  well.  Even  if  a  task  analysis  and  learning  objective 
categorization  has  been  carried  out,  there  can  be  problems  because  of  the 
pitfalls  inherent  in  task  description.  Some  tasks  are  not  readily  described, 
especially  those  involving  a  significant  amount  of  mental  skill.  Many  task 
analyses  will  not  reach  the  correct  level  of  detail. 

If  the  user  has  conscientiously  and  skillfully  carried  out  media  analysis 
and  selection  (and  the  implied  ISD  process)  all  of  the  information  necessary 
for  a  TDNS  will  be  available.  Any  weaknesses  in  the  underlying  analyses, 
however,  will  be  promulgated  into  the  device  development  process  through  the 
TDNS.  Although  there  are  sufficient  safeguards  and  checks  in  the  system  to 
insure  that  a  simulator  or  training  device  is  necessary,  at  this  point  in 
the  process  it  is  difficult  to  determine  if  aVj_  tasks  to  be  trained  in  fact 
require  a  device. 

TDLOA/TDR/TDLR 

These  documents  are  grouped  together  because  they  describe  device  character¬ 
istics,  whether  the  result  of  OWG  discussions  or  prototype  development  and 
test.  A  major  shortcoming  in  the  preparation  of  these  documents  is  that  key 
members  of  the  JWG  lack  specific  guidelines  for  the  types  of  decisions  which 
they  must  make. 


The  JWG  format  was  born  out  of  the  necessity  to  bring  several  different 
kinds  of  expertise  to  bear  on  the  training  device  development  process. 

However,  the  group  format  alone  does  not  insure  effective  interaction 
among  the  users,  training  support  personnel,  and  engineers.  Each  representa¬ 
tive  has  his  own  biases  and  requirements  and  a  language  for  describing  what 
is  wanted  or  needed.  Two  "decision  rules,"  typically  of  the  user,  illus¬ 
trate  a  costly  bias: 

o  In  the  absence  of  knowledge  go  to  a  higher  fidelity,  and 
o  spend  available  money  on  fidelity  until  it  is  exhausted. 

Fidelity  decisions  are  driven  by  a  desire  first  to  use  the  actual  equipment. 
Such  a  perspective  also  tends  to  ignore  consideration  of  instructional 
features. 

Group  problem-solving  is  complicated  enough,  but  when  individuals  representing 
different  scientific,  engineering  and  technical  disciplines  compose  the  group 
the  task  difficulty  is  compounded.  The  lack  of  a  common  conceptual  framework 
and  language  for  describing  training  simulator  configuration  and  capabilities 
is  the  most  significant  impediment  to  intra-JWG  communication.  Improved  com¬ 
munication  among  the  JWG  participants  is  necessary  to  insure  that  training 
requirements  are  efficiently  translated  into  training  simulator  characteristics 
and  ultimately  engineering  specifications. 

Specification  for  RFP 

Once  the  training  characteristics  and  capabilities  of  a  device  have  been 
specified,  PM  TRADE  has  the  task  of  writing  an  engineering  specification  as 
part  of  a  statement  of  work.  This  is  a  critical  task  in  the  training  device 
development  process  and  one  that  is  difficult  to  do  well.  From  the  point  of 
view  of  the  engineer,  he  must  preserve  the  integrity  of  the  training  require¬ 
ments  while  at  the  same  time  detailing  hardware,  software,  and  interface  capa¬ 
bilities.  From  the  point  of  view  of  the  device  manufacturer,  he  must  determine 
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how  to  implement  the  desired  capabilities.  Often  times  the  same  specification 
will  be  viewed  as  too  detailed  by  one  manufacturer  and  not  detailed  enough  by 
another;  the  same  can  also  be  true  for  different  parts  of  the  same  specifica¬ 
tion.  From  the  point  of  view  of  the  user,  he  must  evaluate  the  engineering 
specification  and  determine  if  the  resultant  device  will  meet  his  training 
needs. 

Standardization  is  coming  to  the  process  of  preparing  and  documenting  engineer¬ 
ing  specifications,  but  the  pace  of  standardization  must  be  quickened.  The 
lack  of  a  common,  universally  accepted  (i.e.,  DoD-wide)  framework  for  the 
language  and  format  of  training  device  engineering  specifications  is  a  short¬ 
coming  that  must  be  resolved. 

A  RECOMMENDED  STUDY  APPROACH 

The  goal  of  study  in  this  topic  area  should  be  to  elucidate  the  JWG  decision¬ 
making  process  and  to  develop  a  framework  for  continued  data  collection.  The 
objective  of  this  line  of  research  should  be  to  determine  the  required  for¬ 
mat  and  content  for  guidance  in  the  training  simulator  development  process. 

The  method  should  consist  of  observation  of  JWGs  concerned  with  the  develop¬ 
ment  of  at  least  two,  preferably  three,  different  training  simulators  at 
different  points  in  the  development  cycle.  Data-gathering  should  be  through 
note-taking  and,  when  possible,  follow-up  interviews.  The  observer  should 
not  participate  in  the  proceedings. 

The  analysis  of  the  data  should  focus  on  the  evaluation  of  "critical  incidents." 
That  is,  those  points  of  contention  that  in  the  judgment  of  the  observer  were 
not  easily  resolved.  The  incidents  should  be  categorized  and  described.  Par¬ 
ticular  emphasis  should  be  placed  upon  the  identification  of  the  underlying 
cause  of  the  incident.  For  example,  was  pertinent  data  unknown?  unavailable? 
misinterpreted?  Was  sufficient  detail  available  in  task  analyses?  And  so  on. 
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The  results  of  this  study  would  form  the  basis  for  asking  more  specific 
questions  about  the  JWG  decision  process  and  for  designing  field  studies  to 
answer  them. 
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APPENDIX  B 

WORKSHOP  DETAILS 
o  Agenda 

o  Topic  Area  Questions 
o  Discussion  Summary 
o  Attendees 
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WORKSHOP  AGENDA 


"RESEARCH  ISSUES  IN  THE  DETERMINATION  OF 
SIMULATOR  FIDELITY" 

23-24  July  1981 


23  JULY 

0745 

Coffee  and  Registration 

0830 

Call  to  Order 

John  Brock,  Honeywell  SRC 

0835 

AR1  Welcome 

Col  Franklin  Hart,  Commander,  ARI 

Dr.  Joseph  Zeidner,  Technical 
Director,  ARI 

0840 

ARI  Simulation  Research  Program 

Dr.  Angelo  Mirabella,  ARI 

0850 

SIMTRAIN 

Dr.  Stephen  Hirshfeld,  Honeywell 

0900 

Fidelity  Determi nation--Task 
Objectives  and  Structure 

Dr.  David  Baum,  Honeywell  SRC 

0915 

Workshop  Objectives  and  Organization 

Dr.  Robert  Hays,  ARI 

John  Brock,  Honeywell  SRC 

0925 

Issues  in  Simulation 

Dr.  Robert  Hennessy,  National 
Research  Council 

0940 

Constraints  on  Fidelity 

Maj  Jack  Thorpe,  DARPA 

0955 

A  Fidelity  Definition  and 

Interactive  Factors 

Dr.  Robert  Hays,  ARI 

1010 

Break 

1030 

What  We  Know  and  Who  Should  Know  It 

Clarence  Semple,  Canyon  Research 

1045 

Effectiveness  Issues 

Rohn  Hritz,  Applied  Science  Assoc 

1100 

Definitions  in  Simulation 

Worth  Scanland,  CNET 

1115 

Is  Simulator  Fidelity  the  Question? 

William  Montague,  NPRDC 

1130 

Simulator  Use  in  4  Functional  Areas 

Dr.  Leon  Nawrocki  ,  ARI 

1145 

Working  Group  Organization  and 
Assignments 

Dr.  Robert  Hays,  ARI 

1200 

Lunch 

1300 

Working  Group  Session  1 

1545 

Break 

1640 

Working  Group  Session  II 

1730 

End  of  First  Day 
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WORKSHOP  AGENDA 


"RESEARCH  ISSUES  IN  THE  DETERMINATION  OF 
SIMULATOR  FIDELITY" 

23-24  July  1981 


24  JULY 

0800 

Coffee  and  Rolls 

0830 

Working 

Group  Session 

III 

1000 

Break 

1015 

Working 

Group  Session 

IV 

1145 

Lunch 

1245 

Informal 

Leaders 

Information  Exchange/Topic 
Prepare  Summations 

1345 

Topic  Leaders  Present 

Summati ons 

1500 

Break 

1515 

Closing 

Observati ons 

1545 

Closing 

Remarks 

Dr.  Jesse  Orlansky,  IDA 
Dr.  David  Baum,  Honeywell  SRC 

Dr.  Robert  Hays ,  ARI 
John  Brock,  Honeywell  SRC 
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I  Ef fectiveness  Topics 

A.  Training  Effectiveness  vs.  Transfer  Effectiveness 

Is  there  a  relationship  between  the  scores  of  trainees  on  a  training  device/ 
simulator  (training  effectivenss)  and  their  scores  on  the  actual  equipment 
(transfer  effectiveness)?  If  so:  (1)  Bow  do  we  determine  which  scores  or  com¬ 
bination  of  scores  to  use  in  evaluating  a  training  device?  (2)  How  do  we 
determine  the  nature  of  the  relationship?  (3)  How  might  we  apply  test  development 
methodology  to  develop  reliable  and  valid  evaluation  tests? 


B.  Cost  Effectiveness 

(1)  What  is  cost  effectiveness? 

(2)  How  do  we  measure  it?  Is  there  a  cost  effectiveness  metric? 

(3)  How  does  cost  effectiveness  relate  to  training? 


II  Fidelity  Topics 

A.  Measurement  of  Fidelity 

(1)  What  is  Fidelity? 

(2)  Is  it  measureable?  If  so,  what  kind  of  a  metric?  Why  measure  it? 

(3)  What  is  the  relationship,  if  any,  between  fidelity  and  training? 


B.  Generalizabil ity  of  Flight  Simulation  Data  to  other  Areas  of 

Simulation 

Giver,  that  most  data  on  the  relationship  between  training  simulator 
fidelity  and  effective  training  has  come  from  the  realm  cf  flight  simulation: 

(1)  Can  we  generalize  flight  simulation  data  to  other  areas  of 
simulation  training? 

(2)  What  are  the  components/elements  factors  that  influence 
generalizabil ity? 

(3)  What  are  the  conditions  under  which  generalizability  is  possible? 

(4)  How  do  we  use  the  flight  simulation  data  that  is  generalizable? 
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Ill  Guidance  Topics 

A.  Development  of  a  Format  for  Fidelity  Decision  Making  Guidance 

Given  that  the  persons  who  must  specify  the  characteristics  to  be 
incorporated  into  a  training  device  need  guidance  in  making  fidelity  decisions: 

(1)  On  what  issues/factors  dp  they  need  this  guidance? 

(2)  Is  guidance  available? 

(3)  In  what  format  should  this  guidance  be  provided? 


B.  Impact  of  New  Technologies  on  Fidelity  Decisions 

Given  that  new  technologies  are  being  developed  which  could  impact  on 
training  strategies: 

(1)  How  well  do  we  use  new  technologies? 

(2)  How  can  we  better  incorporate  new  technologies  into  training  systems? 

(3)  Is  is  possible/desirable  for  training  devices  to  keep  up  with  new 
technologies? 


IV  Priorities  and  Support  Topics 

A.  Ranking  Fidelity  Research  Issues  in  terms  of  Necessary  Resources 

vs.  Payoffs 

Given  that  there  are  innumerable  research  issues  relating  to  training 
simulator  fidelity: 

(1)  How  do  we  determine  which  issues  to  address  and  the  order  in  which 
to  address  them? 

(2)  Do  different  groups  rank  issues  differently?  In  which  ways  do  they 
differ?  Why? 


B.  Methods  for  Generating  Long-Term  Tri-Service  Research  Support  and 
Communication 

How  do  we  gain  the  necessary  support  to  undertake  a  long-term  research 
effort  to  answer  the  questions  raised  in  this  workshop? 

Support  is  necessary  from  the  user  level  through  DOD. 
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The  format  chosen  for  the  working  group  sessions  was  successful  in 
stimulating  discussions  on  the  four  topic  areas.  A  wide  diversity  of 
views  was  expressed  as  to  the  appropriateness  or  form  of  the  specific 
questions  assigned  to  each  topic  area.  Indeed*  a  large  portion  of 
Session  I  for  all  topics  was  devoted  to  refining  or  restating  the 
questions.  Therefore,  not  all  of  the  questions  posed  at  the  outset 
of  the  session  were  discussed  or  answered.  In  some  cases  better  (i.e.,  more 

meaningful)  questions  were  the  result. 

In  what  follows  we  enumerate  the  main  points  made  within  each  topic 
area.  These  points  generally  represent  a  consensus  of  opinion,  but 
where  significant  minority  views  were  discussed  they  are  noted  as  well. 

EFFECTIVENESS 

1.  It  proved  difficult  to  establish  general  guidelines  for  defining 
measures  of  device  training  effecti vneess .  The  particular 
measure  chosen,  whether  normative  (e.g.,  comparing  a  conventional 
training  method  with  an  experimental  method)  or  criterion- 
referenced,  depends  upon  the  goals  of  the  evaluation  and  of  the 
particular  training. 

2.  Are  we  concerned  with  terminal  performance  in  a  school  setting 
or  on-the-job?  In  a  military  context  criterion-referenced 
performance,  that  is,  ultimate  performance  on-the-job,  is  of 
paramount  importance.  A  recognition  of  distinction  between 
combat  and  peacetime  performance  of  tasks  is  critical. 

3.  We  must  be  able  to  measure  effectiveness  validly  and 
reliably,  before  we  can  predict  it.  We  do  not  always  have 
the  capability  to  measure  performance,  nor  do  we  always  know 
what  to  measure. 
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4.  It  is  not  possible  to  obtain  combat  performance  measures  in 
peacetime.  We  must  test  what  we  can  test  well  and  generalize 
or  make  assumptions  based  on  sound  psychological  principles 
that  other  skills  are  trained  well.  Although  there  was  general 
agreement  that  a  transfer  of  training  paradigm  is  the  most 
appropriate  for  effectiveness  measurement,  there  was  a  significant 
dissenting  opinion  that  transfer,  per  $e,  does  not  allow  one  to 
isolate  device  effects. 

5.  Subject  matter  experts  (SMEs)  provide  a  valuable  input  on 
performance  effectiveness,  but  we  need  better  methods  for 
obtaining  information  from  SMEs. 


FIDELITY 

1.  There  was  much  discussion  about  whether  we  need  the  concept  of 
fidelity  at  all.  To  make  that  determination  the  groups  attempted 
to  find  a  definition  of  fidelity  that  is  task-independent.  There 
seemed  to  be  broad  agreement  regarding  the  definition  of  the 
physical  aspect  of  fidelity.  There  was  a  consensus  that  (a)  physical 
similarity  was  reasonably  well  understood;  (b)  it  is  important 

for  some  tasks  but  not  for  others;  and  (c)  that  non-physical 
similarity  is  often  more  important  for  training  purposes.  Opinions 
regarding  the  definition  of  non-physical  (i.e.,  the  other  aspect  of 
fidelity)  were  much  more  diverse  and  seemed  to  vary  with  the  type 
of  task  under  consideration.  Major  task  types  identified  were 
procedural,  control,  and  cognitive. 

2.  There  was  a  consensus  that  fidelity  is  at  best  only  one  determinant 
of  training  effectiveness.  Fidelity  is  a  byproduct  of  the  specifica¬ 
tion  of  the  cues  and  responses  required  for  task  knowledge  and 

skill  acqusiition.  This  specification  should  result  from  an  analysis 
of  what  is  to  be  learned. 
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3.  One  can  simulate  an  environment,  equipment,  tasks,  the  behavior 
of  experts,  and  so  on.  Required  fidelity  depends  in  a  complex 
way  on  the  object  of  simulation  and  the  training  purpose. 

4.  It  was  pointed  out  that  the  instructors'  needs  are  often  ignored 

in  a  "fidelity"  analysis.  Instructor  stations,  or  support  functions, 
are  either  left  out  or  poorly  human  factored. 

5.  A  metric  of  fidelity  should  capture  the  categorical,  hierarchical 
nature  of  training  applications  to  particular  tasks.  Fidelity 
cannot  be  measured  on  a  linear  scale. 

GUIDANCE 

1.  The  idea  of  providing  guidance  on  fidelity  decisions,  per  se,  was 
rejected  in  favor  of  an  approach  emphasizing  a  general  class  of 
design  decisions.  Devices  must  be  designed  with  respect  to 
training  goals,  including  the  nature  of  the  tasks  to  be  trained, 
personnel  to  be  trained,  level  of  proficiency  needed,  training 
context  and  methods,  and  so  on.  This  general  framework  is  not 
new  and  utilizing  it  in  design  avoids  problems  associated  with 
focusing  on  only  one  aspect  of  the  problem,  namely,  what  level 

of  fidelity  to  provide.  A  particularly  interesting  design  factor 
that  was  suggested  concerns  the  effectiveness  evaluation  criteria 
of  a  device;  th,-s  was  dubbed  a  "back-end"  analysis.  Attempts 
could  be  made  to  determine  what  is  required  for  a  device  to  be 
successful  in  the  field  and  this  information  would  constitute 
design  criteria. 

2.  In  the  context  of  the  device  design  decision  process,  fidelity  is 

a  byproduct  of  the  training  requirements  analysis.  It  may  not  be  useful 
to  examine  fidelity  issues  outside  of  this  context. 

3.  It  was  universally  agreed  that  we  need  better  methods  for 
communication  between  training  psychologists  and  engineers. 

Present  taxonomic  approaches  to  describing  and  defining  tasks/skills 
are  limited  and  unwieldly. 
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4.  The  point  at  which  training  requirements  are  translated  into 
engineering  requirements  is  critical.  There  was  much  discussion 
about  what  format  was  best  for  presentation  of  training 
information  to  engineers.  A  good  example  of  the  dilemma  is 
whether  it  should  be  in  terms  of  the  number  of  edges  and  corners 
for  a  visual  display  or  in  terms  of  the  device  being  able  to 
achieve  50*  transfer.  However,  psychologists  cannot  tell  engineers 
how  many  edges  and  corners  are  necessary  without  knowing  the 
types  of  discriminations  to  be  learned  and  a  level  of  transfer 

of  training  cannot  be  specified  without  a  complete  account  of 
performance  measurement  on  the  discrimination  tasks. 

5.  Case  studies  were  recomnended  as  an  approach  for  building  a 
guidance  data  base.  Documentation  of  lessons  learned  from 
device  design  decisions,  utilization  patterns  and  success/failure 
of  specific  features  was  encouraged. 

6.  Organizational  changes  were  discussed  as  a  key  method  to 
improving  "corporate  memory"  which  was  judged  to  have  improved 
somewhat  with  the  experience  of  the  last  several  years. 

Specific  changes  suggested  included: 

o  providing  decision-makers  with  direct  experience  on 
devices  being  used  in  the  field; 

o  providing  engineers  with  support  to  see  devices  they 
have  designed  in  use; 

o  reducing  turnover  rates  for  training  device  developers 
by  making  it  a  career  speciality; 

o  involving  designers  with  engineering  decisions,  through 
procurement  and  several  months  of  field  observation  before 
rotating  back  to  design  additional  devices;  and 

o  gearing  rewards  more  to  device  training  effectiveness 
rather  than  exclusively  to  budget  and  schedule  factors. 
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7.  There  are  many  instances  in  which  new  technology,  in  the  form  of 

an  added  training  capability,  e.g.,  automated  performance  measures, 
freeze  and  replay,  etc.,  has  been  misused  or  unused.  Better 
communication  is  needed  between  designers  and  users,  especially 
instructors  who  were  not  involved  in  the  design  process,  as  to 
the  purpose  and  utilization  of  instructional  support  capabilities. 
Better  methods  for  training  of  new  users/instructors  are  needed. 

8.  More  laboratory  research  is  needed  on  the  utilization  and  utility 
of  new  technologies  before  they  are  implemented  in  device  design. 

9.  There  is  a  lack  of  a  focal  point  for  the  management  of  change. 

A  facilitator,  analogous  to  an  agricultural  extension  agent, 
could  be  responsible  for  transmitting  iaeas  and  successes/ 
failures  between  user  community  and  engineers  and  training 
specialists. 

PRIORITIES  AND  SUPPORT 

The  list  ot  research  issues  identified  is  long  and 

diverse.  To  complicate  matters  further,  issues  were  presented  at  different 
levels  of  abstraction.  Prioritization  of  the  identified  issues  proved 
impossible  during  the  group  process,  but  the  groups  determined  criteria 
that  could  be  used  to  prioritize.  Support  was  discussed  in  terms  of 
two  general  categories:  how  to  generate  dollar  investment  for  the  research 
and  how  to  facilitate  information  dissemination  among  services  and 
disciplines . 

The  issues  that  seemed  to  generate  the  most  discussion  and  general  agree¬ 
ment  were  as  follows: 

1.  Oevelop  a  "media  selection  model"  for  training  simulator  design. 

2.  Redirect  the  focus  of  R&D  and  the  user  community  away  from 
fidelity  towards  training  effectiveness. 

3.  Identify  and  accommodate  individual  differences  in  aptitude, 
ability,  or  skill-level  in  training  simulator  design. 
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4.  Determine  how  to  coordinate  the  development  of  a  training 
simulator's  fidelity  and  instructional  features. 

5.  Develop  or  utilize  performance  measures  more  closely  related  to 
real-world  operational /combat  criteria. 

6.  Develop  techniques  for  training  soldiers  how  to  learn.  Increase 
understanding  of  basic  learning  processes. 

7.  Survey  existing  simulators  to  determine  if  positive  transfer 

is  being  achieved.  To  what  degree?  Can  the  relationship  between 
device  fidelity,  features,  and  utilization  and  transfer  be 
quantified?  Over  which  factors  does  the  device  designer  have 
control ? 

8.  Develop  a  better  model  of  the  factors  impacting  device 
utilization.  What  are  the  correlates  of  user  acceptance? 

Of  increased  student  motivation?  How  do  we  cope  with  instructor 
turnover;  training  the  instructor  how  to  use  the  device? 

9.  To  what  extent  do  mental  models  underly  task  performance? 

What  methods  can  be  used  to  facilitate  the  acquisition  of 
mental  models? 

10.  Develop  a  technology  for  field-based  experimental  evaluations 
of  training  effectiveness.  How  do  we  intervene  in  ongoing 
curricula  with  new  simulation  ideas? 

11.  How  do  we  promote/ facilitate  coronuni cation  between  the 
psychologist  and  the  engineer?  What  are  the  job  qualifications 
for  a  composite  behavioral  scientist/engineer? 

The  criteria  against  which  issues  could  be  prioritized  included: 

1 .  cost  of  the  research 

2.  likelihood  of  effective  payoff 

3.  feasibility  of  doing  the  research 

4.  sponsor  priorities  (i.e.,  policy) 
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5.  technological  gap  closure 

6.  general izability  of  the  outcome 

7.  user  acceptance 

8.  training  effectiveness  potential 

9.  availability  of  other  necessary  resources 

10.  sex  appeal/PR  value/visibility 

11.  operational  requirements 

12.  low  life-cycle  cost  of  product  of  research 

A  peripheral  issue  involved  the  categorization  of  R&D  and  whether  it  is 
funded  with  6.1,  6.2,  or  6.3  monies:  How  can  we  better  match  research 
that  is  justified  with  the  dollars  available? 

The  support  issues,  as  described,  concerned  resource  allocation  and 
dissemination  of  information. 

1.  Industry  IR&D  should  be  considered  as  a  potential  source  of 
support  despite  the  fact  that  companies  will  be  reluctant 

to  put  money  into  endeavors  that  might  benefit  their  competition. 

2.  More  concrete  evidence  of  the  impact  and  payoff  of  training 
research  is  needed;  especially  in  terms  of  benefits  for  the  user 
community.  The  cost  of  training  research  could  be  put  into  the 
development  costs  of  an  operational  system  regardless  of  who 
paid  for  it.  This  could  save  money  in  the  long  run. 

3.  Application  specialists  (like  Agriculture  Field  Service 
Representatives)  for  training  could  be  established.  They 
could  facilitate  during  both  the  design  and  utilization 
phases  of  a  training  simulator’s  life  cycle. 
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4.  Remove  limitations  on  the  public  distribution  of  Government 
studies  to  make  literature  more  widely  available. 

5.  Establish  a  centralized  automated  data  base  on  training 
effectiveness.  Such  information  as  who  is  designing/building 
what  devices,  simulator  descriptions,  fidelity  references,  types 
of  tasks  trained,  demonstrated  transfer,  etc.  would  be  stored. 
The  maintenance,  update,  and  interpretation  of  such  a  data  base 
would  be  difficult  and  great  care  would  have  to  be  taken  in 
designing  and  implementing  it. 
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